Skip to content
LLM Atlas

Benchmarks

WinoGrande

saturatedReasoningcontamination risk: highlaunched 2019-07-24

Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.

What it measures

Binary fill-in-the-blank commonsense problems, scaled from Winograd schemas and adversarially filtered to reduce dataset shortcuts.

Primary metric
accuracy (%)
Judge model
not judged by a model
Human baseline
94Crowd-worker estimate reported by the original paper; compare with care because current lm-eval runs use the validation split rather than the hidden-label official test set.
Languages
English

Health

On the 1,267-example validation split, a 50% accuracy estimate has an approximate 95% binomial margin of error of +/-2.8 points; paired predictions are preferable for close model comparisons.

Status evidence
The original XL result was 79.1% against a 94.0% human estimate, leaving real headroom, but the static public inputs and widespread validation-set use make modern scores contamination-sensitive.
Saturated since
2022-02-17
Refresh cycle
static
Test set public
yes

Official test labels remain hidden, and lm-evaluation-harness exposes a decontamination query, but the test inputs and the fully labeled validation split are public and unchanged.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Independently measured

ModelScoreHarnessMeasuredProvenanceHealth
LLaMA-3.1 405B BaseMeta AI89.2%not reported2024-07-23ind.saturatedcontaminated
GPT-4OpenAI87.5%not reported2023-03-14ind.saturatedcontaminated
Falcon 180BTechnology Innovation Institute87.1%not reported2023-09-06ind.saturatedcontaminated
DeepSeek-V2 BaseDeepSeek86.3%not reported2024-05-07ind.saturatedcontaminated
DeepSeek-V3DeepSeek85.2%not reported2024-12-26ind.saturatedcontaminated
PaLM 540BGoogle Research85.1%not reported2022-04-04ind.saturatedcontaminateddisputed
PaLM 540BGoogle Research83.7%not reported2022-04-04ind.saturatedcontaminateddisputed
DS-Coder-V2-BaseDeepSeek83.7%not reported2024-06-17ind.saturatedcontaminated
LLaMA 3 70BMeta AI83.5%not reported2024-04-18ind.saturatedcontaminated
Qwen2.5 72B BaseAlibaba82.3%not reported2024-09-19ind.saturatedcontaminated
LLaMA 3 405BMeta AI82.2%not reported2024-07-23ind.saturatedcontaminated
GPT-3.5OpenAI81.6%not reported2022-03-15ind.saturatedcontaminated
Phi-3-smallMicrosoft81.5%not reported2024-04-23ind.saturatedcontaminated
Phi-3-mediumMicrosoft81.5%not reported2024-04-23ind.saturatedcontaminated
PaLM 540BGoogle Research81.1%not reported2022-04-04ind.saturatedcontaminateddisputed
Qwen2.5-Coder-32BAlibaba80.8%not reported2024-09-18ind.saturatedcontaminated
LLAMA 2 70BMeta AI80.2%not reported2023-07-18ind.saturatedcontaminated
GLaMGoogle79.2%not reported2021-12-13ind.saturatedcontaminateddisputed
Gemma 7BGoogle DeepMind79.0%not reported2024-02-21ind.saturatedcontaminateddisputed
MT-NLGMicrosoft,Nvidia78.9%not reported2021-10-11ind.saturatedcontaminateddisputed
Falcon2-11B stage 4Technology Innovation Institute78.3%not reported2024-05-09ind.saturatedcontaminated
Nemotron-4 15BNVIDIA78.0%not reported2024-02-26ind.saturatedcontaminated
GPT-3OpenAI77.7%not reported2022-01-27ind.saturatedcontaminateddisputed
Mixtral 8x7BMistral AI77.2%not reported2023-12-11ind.saturatedcontaminateddisputed
LLaMA 65BMeta AI77.0%not reported2023-02-24ind.saturatedcontaminated
Falcon 40BTechnology Innovation Institute76.9%not reported2023-03-15ind.saturatedcontaminateddisputed
LLAMA 2 34BMeta AI76.7%not reported2023-07-18ind.saturatedcontaminated
Falcon 40BTechnology Innovation Institute76.4%not reported2023-03-15ind.saturatedcontaminateddisputed
LLaMA 33BMeta AI76.0%not reported2023-02-24ind.saturatedcontaminated
LLaMA 3 8BMeta AI75.7%not reported2024-04-18ind.saturatedcontaminated
Mistral 7BMistral AI75.3%not reported2023-09-27ind.saturatedcontaminateddisputed
Chinchilla 70BDeepMind74.9%not reported2022-03-29ind.saturatedcontaminated
Mistral 7BMistral AI74.2%not reported2023-09-27ind.saturatedcontaminateddisputed
MT-NLGMicrosoft,Nvidia73.7%not reported2021-10-11ind.saturatedcontaminateddisputed
GLaMGoogle73.5%not reported2021-12-13ind.saturatedcontaminateddisputed
phi-1.5Microsoft73.4%not reported2023-09-11ind.saturatedcontaminated
GPT-3OpenAI73.2%not reported2022-01-27ind.saturatedcontaminateddisputed
MT-NLGMicrosoft,Nvidia73.0%not reported2021-10-11ind.saturatedcontaminateddisputed
LLaMA 13BMeta AI73.0%not reported2023-02-24ind.saturatedcontaminated
GLaMGoogle73.0%not reported2021-12-13ind.saturatedcontaminateddisputed
Qwen2.5-Coder-7BAlibaba72.9%not reported2024-09-18ind.saturatedcontaminated
LLaMA-2 13BMeta AI72.8%not reported2023-07-18ind.saturatedcontaminated
Gemma 7BGoogle DeepMind72.3%not reported2024-02-21ind.saturatedcontaminateddisputed
Yi 6B01.AI71.3%not reported2023-11-02ind.saturatedcontaminated
MPT 30BMosaicML71.0%not reported2023-06-22ind.saturatedcontaminated
Phi-3-miniMicrosoft70.8%not reported2024-04-23ind.saturatedcontaminated
GPT-3OpenAI70.2%not reported2022-01-27ind.saturatedcontaminateddisputed
GopherDeepMind70.2%not reported2021-12-08ind.saturatedcontaminateddisputed
GopherDeepMind70.1%not reported2021-12-08ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI70.1%not reported2023-02-24ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI69.6%not reported2023-02-24ind.saturatedcontaminateddisputed
LLAMA 2 7BMeta AI69.2%not reported2023-07-18ind.saturatedcontaminateddisputed
LLAMA 2 7BMeta AI69.1%not reported2023-07-18ind.saturatedcontaminateddisputed
GPT-3.5 version 1106OpenAI68.8%not reported2023-11-06ind.saturatedcontaminated
MPT 7BMosaicML68.6%not reported2023-05-05ind.saturatedcontaminateddisputed
MPT 7BMosaicML68.3%not reported2023-05-05ind.saturatedcontaminateddisputed
MPT 7BMosaicML68.0%not reported2023-05-05ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute67.2%not reported2023-04-24ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute67.2%not reported2023-04-24ind.saturatedcontaminateddisputed
LLaMA 7BMeta AI66.9%not reported2023-02-24ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute66.3%not reported2023-04-24ind.saturatedcontaminateddisputed
Falcon 7BTechnology Innovation Institute66.2%not reported2023-04-24ind.saturatedcontaminateddisputed
GPT-neox-20BEleutherAI66.1%not reported2022-04-07ind.saturatedcontaminated
INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI65.8%not reported2024-11-29ind.saturatedcontaminated
Gemma 2BGoogle DeepMind65.4%not reported2024-02-21ind.saturatedcontaminated
Llama-3-In 8bMeta AI65.0%not reported2024-04-18ind.saturatedcontaminated
XGen-7BSalesforce64.9%not reported2023-06-27ind.saturatedcontaminated
GPT-J-6BEleutherAI,LAION64.5%not reported2021-08-05ind.saturatedcontaminated
StarCoder2-15BHugging Face,ServiceNow,NVIDIA,BigCode64.3%not reported2024-02-20ind.saturatedcontaminated
DS-Coder-33B-BaseDeepSeek,Peking University62.0%not reported2023-11-02ind.saturatedcontaminated
Mixtral 8x7BMistral AI62.0%not reported2023-12-11ind.saturatedcontaminateddisputed
Dolly-v2-12BDatabricks61.8%not reported2023-04-11ind.saturatedcontaminated
OPT-1.3BMeta AI61.0%not reported2022-05-11ind.saturatedcontaminated
Cerebras-GPT-13BCerebras Systems60.8%not reported2023-03-20ind.saturatedcontaminated
Qwen2.5-Coder-1.5BAlibaba60.7%not reported2024-09-18ind.saturatedcontaminated
GPT2-XL-1.5BOpenAI58.3%not reported2019-11-05ind.saturatedcontaminated
DS-Coder-6.7B-BaseDeepSeek,Peking University57.6%not reported2023-11-02ind.saturatedcontaminated
StarCoder2-3BHugging Face,ServiceNow,NVIDIA,BigCode57.1%not reported2024-02-22ind.saturatedcontaminated
StarCoder2-7BHugging Face,ServiceNow,NVIDIA,BigCode57.1%not reported2024-02-20ind.saturatedcontaminated
Gemma 7BGoogle DeepMind55.6%not reported2024-02-21ind.saturatedcontaminateddisputed
Phi-2Microsoft54.7%not reported2023-12-12ind.saturatedcontaminated
Mistral 7BMistral AI54.2%not reported2023-09-27ind.saturatedcontaminateddisputed
DS-Coder-1.3B-BaseDeepSeek,Peking University53.3%not reported2023-11-02ind.saturatedcontaminated

Vendor-reported

Reported by the model's own maker. No independent measurement of these numbers is on this site.

ModelScoreHarnessMeasuredProvenanceHealth
Claude 3 OpusAnthropic88.5%not reported2024-02-29vendorsaturatedcontaminatedvendor only
Claude 3 SonnetAnthropic75.1%not reported2024-02-29vendorsaturatedcontaminatedvendor only
Claude 3 HaikuAnthropic74.2%not reported2024-03-07vendorsaturatedcontaminatedvendor only

Trajectory

Best score measured on this site

0255075100GPT2-XL-1.5B: 58.3%GPT-J-6B: 64.5%MT-NLG: 78.9%MT-NLG: 73.7%MT-NLG: 73.0%Gopher: 70.2%Gopher: 70.1%GLaM: 79.2%GLaM: 73.5%GLaM: 73.0%GPT-3: 77.7%GPT-3: 73.2%GPT-3: 70.2%GPT-3.5: 81.6%Chinchilla 70B: 74.9%PaLM 540B: 85.1%PaLM 540B: 83.7%PaLM 540B: 81.1%GPT-neox-20B: 66.1%OPT-1.3B: 61.0%LLaMA 65B: 77.0%LLaMA 33B: 76.0%LLaMA 13B: 73.0%LLaMA 7B: 70.1%LLaMA 7B: 69.6%LLaMA 7B: 66.9%GPT-4: 87.5%Falcon 40B: 76.9%Falcon 40B: 76.4%Cerebras-GPT-13B: 60.8%Dolly-v2-12B: 61.8%Falcon 7B: 67.2%Falcon 7B: 67.2%Falcon 7B: 66.3%Falcon 7B: 66.2%MPT 7B: 68.6%MPT 7B: 68.3%MPT 7B: 68.0%MPT 30B: 71.0%XGen-7B: 64.9%LLAMA 2 70B: 80.2%LLAMA 2 34B: 76.7%LLaMA-2 13B: 72.8%LLAMA 2 7B: 69.2%LLAMA 2 7B: 69.1%Falcon 180B: 87.1%phi-1.5: 73.4%Mistral 7B: 75.3%Mistral 7B: 74.2%Mistral 7B: 54.2%Yi 6B: 71.3%DS-Coder-33B-Base: 62.0%DS-Coder-6.7B-Base: 57.6%DS-Coder-1.3B-Base: 53.3%GPT-3.5 version 1106: 68.8%Mixtral 8x7B: 77.2%Mixtral 8x7B: 62.0%Phi-2: 54.7%StarCoder2-15B: 64.3%StarCoder2-7B: 57.1%Gemma 7B: 79.0%Gemma 7B: 72.3%Gemma 2B: 65.4%Gemma 7B: 55.6%StarCoder2-3B: 57.1%Nemotron-4 15B: 78.0%LLaMA 3 70B: 83.5%LLaMA 3 8B: 75.7%Llama-3-In 8b: 65.0%Phi-3-small: 81.5%Phi-3-medium: 81.5%Phi-3-mini: 70.8%DeepSeek-V2 Base: 86.3%Falcon2-11B stage 4: 78.3%DS-Coder-V2-Base: 83.7%LLaMA-3.1 405B Base: 89.2%LLaMA 3 405B: 82.2%Qwen2.5-Coder-32B: 80.8%Qwen2.5-Coder-7B: 72.9%Qwen2.5-Coder-1.5B: 60.7%Qwen2.5 72B Base: 82.3%INTELLECT-1-Instruct: 65.8%DeepSeek-V3: 85.2%GPT2-XL-1.5B: 58.3%, best at the timeGPT-J-6B: 64.5%, best at the timeMT-NLG: 78.9%, best at the timeGLaM: 79.2%, best at the timeGPT-3.5: 81.6%, best at the timePaLM 540B: 85.1%, best at the timeGPT-4: 87.5%, best at the timeLLaMA-3.1 405B Base: 89.2%, best at the time20192024
Each dot is a score measured on this site; the line steps up at each new best and stays flat until the next one. A line that has stopped climbing is what saturation looks like.

As benchwiki records it

human baselineBERT-Large (fine-tuned) — 64.9 on 2019-07-24 (independent)RoBERTa-large (fine-tuned) — 79.1 on 2019-07-24 (independent)GPT-3 175B (few-shot) — 77.7 on 2020-07-22 (vendor-reported)PaLM 540B (5-shot) — 85.1 on 2022-04-04 (vendor-reported)GPT-4 (5-shot) — 87.5 on 2023-03-14 (vendor-reported)Llama 2 70B (5-shot) — 81.8 on 2023-07-18 (vendor-reported)Claude 3 Opus (5-shot) — 88.5 on 2024-03-04 (vendor-reported)Llama 3 70B (5-shot) — 83.1 on 2024-04-18 (vendor-reported)Llama 3.1 405B (5-shot) — 89.2 on 2024-07-23 (vendor-reported)20192024940
independent · vendor-reported. Points are not joined: the scores come from different models, harnesses and protocol versions, and a line between them would claim a continuity the data does not have.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

Source

This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2026-07-25; fetched 2026-10-19 by this site.