Skip to content
LLM Atlas

Benchmarks

HellaSwag

saturatedReasoningcontamination risk: highlaunched 2019-05-19

Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.

What it measures

Choose the plausible continuation of an everyday event from one real ending and three adversarially filtered machine generations.

Primary metric
normalized accuracy (%)
Judge model
not judged by a model
Human baseline
95.6Overall test accuracy reported by the authors; random choice among four endings is 25%.
Languages
English

Health

On 10,003 independent test items, a score near 50% has an approximate 95% binomial margin of error of +/-1.0 point; near 95% it is about +/-0.4 point. Prompting and length normalization can move results by more than this.

Status evidence
GPT-4 base reached 95.3% on a private holdout against the original 95.6% human reference; the official leaderboard closed submissions in 2024, and public validation data is contamination-prone.
Saturated since
2023-03-14
Refresh cycle
static
Test set public
no

Official test labels were withheld and GPT-4 used a private holdout, but the standard harness evaluates the fully public 10,042-item validation split. Report the split and check training overlap.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Independently measured

ModelScoreHarnessMeasuredProvenanceHealth
GPT-4OpenAI95.3%not reported2023-03-14ind.saturatedcontaminated
LLaMA-3.1 405B BaseMeta AI89.2%not reported2024-07-23ind.saturatedcontaminated
Falcon 180BTechnology Innovation Institute89.0%not reported2023-09-06ind.saturatedcontaminated
DeepSeek-V3DeepSeek88.9%not reported2024-12-26ind.saturatedcontaminated
DeepSeek-V2 BaseDeepSeek87.1%not reported2024-05-07ind.saturatedcontaminated
Mixtral 8x7BMistral AI86.7%not reported2023-12-11ind.saturatedcontaminateddisputed
LLAMA 2 70BMeta AI85.3%not reported2023-07-18ind.saturatedcontaminated
Falcon 40BTechnology Innovation Institute85.3%not reported2023-03-15ind.saturatedcontaminateddisputed
Qwen2.5 72B BaseAlibaba84.8%not reported2024-09-19ind.saturatedcontaminated
Mixtral 8x7BMistral AI84.4%not reported2023-12-11ind.saturatedcontaminateddisputed
LLaMA 65BMeta AI84.2%not reported2023-02-24ind.saturatedcontaminated
StableBeluga2 70BStability AI84.1%not reported2023-07-20ind.saturatedcontaminated
PaLM 540BGoogle Research83.8%not reported2022-04-04ind.saturatedcontaminateddisputed
PaLM 540BGoogle Research83.6%not reported2022-04-04ind.saturatedcontaminateddisputed
PaLM 540BGoogle Research83.4%not reported2022-04-04ind.saturatedcontaminateddisputed
Qwen2.5-Coder-32BAlibaba83.0%not reported2024-09-18ind.saturatedcontaminated
Falcon2-11B stage 4Technology Innovation Institute82.9%not reported2024-05-09ind.saturatedcontaminateddisputed
Falcon 40BTechnology Innovation Institute82.8%not reported2023-03-15ind.saturatedcontaminateddisputed
Falcon 40BTechnology Innovation Institute82.8%not reported2023-03-15ind.saturatedcontaminateddisputed
LLaMA 33BMeta AI82.8%not reported2023-02-24ind.saturatedcontaminated
MT-NLGMicrosoft,Nvidia82.4%not reported2021-10-11ind.saturatedcontaminateddisputed
Nemotron-4 15BNVIDIA82.4%not reported2024-02-26ind.saturatedcontaminated
Phi-3-mediumMicrosoft82.4%not reported2024-04-23ind.saturatedcontaminated
Gemma 7BGoogle DeepMind82.2%not reported2024-02-21ind.saturatedcontaminateddisputed
GPT-3.5OpenAI82.2%not reported2022-11-28ind.saturatedcontaminated
Falcon2-11B stage 4Technology Innovation Institute82.1%not reported2024-05-09ind.saturatedcontaminateddisputed
GPT-3.5OpenAI81.5%not reported2022-03-15ind.saturatedcontaminated
Gemma 7BGoogle DeepMind81.2%not reported2024-02-21ind.saturatedcontaminateddisputed
Mistral 7BMistral AI81.0%not reported2023-09-27ind.saturatedcontaminated
LLaMA-2 13BMeta AI80.7%not reported2023-07-18ind.saturatedcontaminated
MT-NLGMicrosoft,Nvidia80.2%not reported2021-10-11ind.saturatedcontaminateddisputed
MT-NLGMicrosoft,Nvidia80.2%not reported2021-10-11ind.saturatedcontaminateddisputed
GPT-3OpenAI79.3%not reported2022-01-27ind.saturatedcontaminateddisputed
GopherDeepMind79.2%not reported2021-12-08ind.saturatedcontaminated
LLaMA 13BMeta AI79.2%not reported2023-02-24ind.saturatedcontaminated
OPT (175B)Meta AI79.1%not reported2022-05-02ind.saturatedcontaminated
GPT-3OpenAI78.9%not reported2022-01-27ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute78.1%not reported2023-04-24ind.saturatedcontaminateddisputed
GPT-3OpenAI78.1%not reported2022-01-27ind.saturatedcontaminateddisputed
davinci (175B)OpenAI77.5%not reportednot reportedind.saturatedcontaminated
GLaMGoogle77.2%not reported2021-12-13ind.saturatedcontaminateddisputed
LLAMA 2 7BMeta AI77.2%not reported2023-07-18ind.saturatedcontaminateddisputed
Phi-3-smallMicrosoft77.0%not reported2024-04-23ind.saturatedcontaminated
GLaMGoogle76.8%not reported2021-12-13ind.saturatedcontaminateddisputed
Qwen2.5-Coder-7BAlibaba76.8%not reported2024-09-18ind.saturatedcontaminated
Phi-3-miniMicrosoft76.7%not reported2024-04-23ind.saturatedcontaminated
GLaMGoogle76.6%not reported2021-12-13ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute76.4%not reported2023-04-24ind.saturatedcontaminateddisputed
MPT 7BMosaicML76.4%not reported2023-05-05ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute76.3%not reported2023-04-24ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI76.2%not reported2023-02-24ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI76.1%not reported2023-02-24ind.saturatedcontaminateddisputed
MPT 7BMosaicML76.1%not reported2023-05-05ind.saturatedcontaminateddisputed
OPT (66B)Meta AI74.5%not reported2022-05-03ind.saturatedcontaminated
BLOOM (176B)Hugging Face,BigScience74.4%not reported2022-07-06ind.saturatedcontaminated
Yi 6B01.AI74.4%not reported2023-11-02ind.saturatedcontaminated
XGen-7BSalesforce74.2%not reported2023-06-27ind.saturatedcontaminated
Falcon 7BTechnology Innovation Institute74.1%not reported2023-04-24ind.saturatedcontaminateddisputed
INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI71.4%not reported2024-11-29ind.saturatedcontaminated
Gemma 2BGoogle DeepMind71.4%not reported2024-02-21ind.saturatedcontaminated
Baichuan2 13BBaichuan70.8%not reported2023-09-06ind.saturatedcontaminated
Dolly-v2-12BDatabricks70.8%not reported2023-04-11ind.saturatedcontaminated
GPT-neox-20BEleutherAI70.5%not reported2022-04-07ind.saturatedcontaminated
curie (6.7B)OpenAI68.2%not reportednot reportedind.saturatedcontaminated
Baichuan2 7BBaichuan68.0%not reported2023-09-20ind.saturatedcontaminated
text-curie-001OpenAI67.6%not reportednot reportedind.saturatedcontaminated
GPT-J-6BEleutherAI,LAION66.2%not reported2021-08-05ind.saturatedcontaminated
Qwen2.5-Coder-1.5BAlibaba61.8%not reported2024-09-18ind.saturatedcontaminated
Cerebras-GPT-13BCerebras Systems59.4%not reported2023-03-20ind.saturatedcontaminated
LLAMA 2 7BMeta AI57.1%not reported2023-07-18ind.saturatedcontaminateddisputed
MPT 7BMosaicML57.1%not reported2023-05-05ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI56.2%not reported2023-02-24ind.saturatedcontaminateddisputed
text-babbage-001OpenAI56.1%not reportednot reportedind.saturatedcontaminated
babbage (1.3B)OpenAI55.5%not reported2020-06-22ind.saturatedcontaminated
Falcon 7BTechnology Innovation Institute54.2%not reported2023-04-24ind.saturatedcontaminateddisputed
Phi-2Microsoft53.6%not reported2023-12-12ind.saturatedcontaminated
phi-1.5Microsoft47.6%not reported2023-09-11ind.saturatedcontaminated
ada (350M)OpenAI43.5%not reportednot reportedind.saturatedcontaminated
text-ada-001OpenAI42.9%not reportednot reportedind.saturatedcontaminated
GPT-Neo-2.7BEleutherAI42.7%not reported2021-03-30ind.saturatedcontaminated
OPT-1.3BMeta AI41.5%not reported2022-05-11ind.saturatedcontaminated
GPT2-XL-1.5BOpenAI40.0%not reported2019-11-05ind.saturatedcontaminated

Trajectory

Best score measured on this site

0255075100GPT2-XL-1.5B: 40.0%babbage (1.3B): 55.5%GPT-Neo-2.7B: 42.7%GPT-J-6B: 66.2%MT-NLG: 82.4%MT-NLG: 80.2%MT-NLG: 80.2%Gopher: 79.2%GLaM: 77.2%GLaM: 76.8%GLaM: 76.6%GPT-3: 79.3%GPT-3: 78.9%GPT-3: 78.1%GPT-3.5: 81.5%PaLM 540B: 83.8%PaLM 540B: 83.6%PaLM 540B: 83.4%GPT-neox-20B: 70.5%OPT (175B): 79.1%OPT (66B): 74.5%OPT-1.3B: 41.5%BLOOM (176B): 74.4%GPT-3.5: 82.2%LLaMA 65B: 84.2%LLaMA 33B: 82.8%LLaMA 13B: 79.2%LLaMA 7B: 76.2%LLaMA 7B: 76.1%LLaMA 7B: 56.2%GPT-4: 95.3%Falcon 40B: 85.3%Falcon 40B: 82.8%Falcon 40B: 82.8%Cerebras-GPT-13B: 59.4%Dolly-v2-12B: 70.8%Falcon 7B: 78.1%Falcon 7B: 76.4%Falcon 7B: 76.3%Falcon 7B: 74.1%Falcon 7B: 54.2%MPT 7B: 76.4%MPT 7B: 76.1%MPT 7B: 57.1%XGen-7B: 74.2%LLAMA 2 70B: 85.3%LLaMA-2 13B: 80.7%LLAMA 2 7B: 77.2%LLAMA 2 7B: 57.1%StableBeluga2 70B: 84.1%Falcon 180B: 89.0%Baichuan2 13B: 70.8%phi-1.5: 47.6%Baichuan2 7B: 68.0%Mistral 7B: 81.0%Yi 6B: 74.4%Mixtral 8x7B: 86.7%Mixtral 8x7B: 84.4%Phi-2: 53.6%Gemma 7B: 82.2%Gemma 7B: 81.2%Gemma 2B: 71.4%Nemotron-4 15B: 82.4%Phi-3-medium: 82.4%Phi-3-small: 77.0%Phi-3-mini: 76.7%DeepSeek-V2 Base: 87.1%Falcon2-11B stage 4: 82.9%Falcon2-11B stage 4: 82.1%LLaMA-3.1 405B Base: 89.2%Qwen2.5-Coder-32B: 83.0%Qwen2.5-Coder-7B: 76.8%Qwen2.5-Coder-1.5B: 61.8%Qwen2.5 72B Base: 84.8%INTELLECT-1-Instruct: 71.4%DeepSeek-V3: 88.9%GPT2-XL-1.5B: 40.0%, best at the timebabbage (1.3B): 55.5%, best at the timeGPT-J-6B: 66.2%, best at the timeMT-NLG: 82.4%, best at the timePaLM 540B: 83.8%, best at the timeLLaMA 65B: 84.2%, best at the timeGPT-4: 95.3%, best at the time20192024
Each dot is a score measured on this site; the line steps up at each new best and stays flat until the next one. A line that has stopped climbing is what saturation looks like.

As benchwiki records it

human baselineBERT-Large — 47.3 on 2019-05-19 (independent)GPT-4 base (10-shot) — 95.3 on 2023-03-14 (vendor-reported)RoBERTa — 85.2 on 2019-07-25 (independent)GPT-3 175B (few-shot) — 79.3 on 2020-07-22 (vendor-reported)PaLM 540B — 83.4 on 2022-04-01 (vendor-reported)LLaMA-65B — 84.2 on 2023-02-27 (vendor-reported)Gemini Ultra (10-shot decontaminated) — 87.8 on 2023-12-06 (vendor-reported)Claude 3 Opus — 95.4 on 2024-03-04 (vendor-reported)Llama 3 70B — 88 on 2024-04-18 (vendor-reported)GPT-4o — 95.3 on 2024-08-01 (vendor-reported)20192024960
independent · vendor-reported. Points are not joined: the scores come from different models, harnesses and protocol versions, and a line between them would claim a continuity the data does not have.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

Source

This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2026-07-25; fetched 2026-10-19 by this site.