Skip to content
LLM Atlas

Benchmarks

MMLU

saturatedKnowledgecontamination risk: highlaunched 2020-09-07

Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.

Superseded by mmlu-pro.

What it measures

57-subject multiple-choice exam spanning STEM, humanities, and social sciences — the defining knowledge benchmark of the 2020–2024 era.

Primary metric
accuracy (%)
Judge model
not judged by a model
Human baseline
89.8Estimated expert-level accuracy from the original paper (per-subject specialists, e.g. 87% for a professional-level test-taker on morality, ~90% averaged). Unspecialized human crowdworkers scored only 34.5%.
Languages
English

Health

14,042 questions → tight CIs (±~0.5pt at 85%), but per-subject splits are small (some subjects ~100 questions → ±~8pt); never compare models on a single subject. Label errors (~2–6%) dwarf sampling error at the frontier.

Status evidence
Frontier models cluster at 86–92% since 2024; ~2–6% label errors put the effective ceiling near 90%, so top-model gaps are within noise.
Saturated since
2024-06-01
Refresh cycle
static
Test set public
yes

None. Test set has been fully public on GitHub/HuggingFace since 2020 and appears verbatim in web-scale crawls; most modern pretraining corpora contain it despite decontamination attempts.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Independently measured

ModelScoreHarnessMeasuredProvenanceHealth
Claude 3.5 Sonnet (20241022)Anthropic87.3%not reported2024-10-22ind.saturatedcontaminatedsuperseded
DeepSeek-V3DeepSeek87.2%not reported2024-12-26ind.saturatedcontaminatedsupersededdisputed
DeepSeek-V3DeepSeek87.1%not reported2024-12-26ind.saturatedcontaminatedsupersededdisputed
Gemini 1.5 Pro (002)Google DeepMind86.9%not reported2024-09-24ind.saturatedcontaminatedsuperseded
Claude 3.5 Sonnet (20240620)Anthropic86.5%not reported2024-06-20ind.saturatedcontaminatedsuperseded
GPT-4OpenAI86.4%not reported2023-03-14ind.saturatedcontaminatedsuperseded
Llama 3.3 70BMeta AI86.3%not reported2024-12-06ind.saturatedcontaminatedsupersededdisputed
Gemini 1.5 Pro (May 2024)Google DeepMind85.9%not reported2024-05-14ind.saturatedcontaminatedsuperseded
Qwen2.5 72BAlibaba85.3%not reported2024-09-19ind.saturatedcontaminatedsupersededdisputed
Qwen2.5 72B BaseAlibaba85.0%not reported2024-09-19ind.saturatedcontaminatedsuperseded
Phi-4 14bMicrosoft Research84.8%not reported2024-12-12ind.saturatedcontaminatedsuperseded
Claude 3 OpusAnthropic84.6%not reported2024-02-29ind.saturatedcontaminatedsuperseded
LLaMA 3 405BMeta AI84.5%not reported2024-07-23ind.saturatedcontaminatedsuperseded
LLaMA-3.1 405B BaseMeta AI84.4%not reported2024-07-23ind.saturatedcontaminatedsuperseded
GPT-4o (2024-08-06)OpenAI84.3%not reported2024-08-06ind.saturatedcontaminatedsuperseded
GPT-4oOpenAI84.2%not reported2024-05-13ind.saturatedcontaminatedsuperseded
Qwen2.5 72BAlibaba83.4%not reported2024-09-19ind.saturatedcontaminatedsupersededdisputed
GPT-4OpenAI82.4%not reported2023-06-13ind.saturatedcontaminatedsuperseded
OpenHands + Qwen-2-72bAlibaba82.4%not reported2024-06-07ind.saturatedcontaminatedsuperseded
Amazon Nova ProAmazon82.0%not reported2024-12-03ind.saturatedcontaminatedsuperseded
Gemini 1.5 Pro (Feb 2024)Google DeepMind81.9%not reported2024-02-15ind.saturatedcontaminatedsuperseded
GPT-4o-miniOpenAI81.8%not reported2024-07-18ind.saturatedcontaminatedsupersededdisputed
GPT-4 Turbo (2024-04-09)OpenAI81.3%not reported2024-04-09ind.saturatedcontaminatedsuperseded
Llama 3.2 Vision Instruct Turbo (90B)Meta AI80.3%not reported2024-09-24ind.saturatedcontaminatedsuperseded
Llama 3.1 70BMeta AI80.1%not reported2024-07-23ind.saturatedcontaminatedsuperseded
Mistral Large 2 (2407)Mistral AI80.0%not reported2024-07-24ind.saturatedcontaminatedsuperseded
Qwen2.5 14b instructAlibaba79.9%not reported2024-09-19ind.saturatedcontaminatedsuperseded
Gemini 2.0 Flash (Experimental)Google DeepMind,Google79.7%not reported2024-12-11ind.saturatedcontaminatedsuperseded
GPT-4 TurboOpenAI79.6%not reported2023-11-06ind.saturatedcontaminatedsuperseded
Yi Large (Preview)01.AI79.3%not reported2024-05-13ind.saturatedcontaminatedsuperseded
Llama 3.3 70BMeta AI79.1%not reported2024-12-06ind.saturatedcontaminatedsupersededdisputed
Qwen2.5-Coder-32BAlibaba79.1%not reported2024-09-18ind.saturatedcontaminatedsupersededdisputed
DeepSeek-V2 BaseDeepSeek78.4%not reported2024-05-07ind.saturatedcontaminatedsuperseded
Phi-3-mediumMicrosoft78.0%not reported2024-04-23ind.saturatedcontaminatedsupersededdisputed
Gemini 1.5 Flash (001)Google DeepMind77.9%not reported2024-05-23ind.saturatedcontaminatedsuperseded
Phi-3-mediumMicrosoft77.9%not reported2024-04-23ind.saturatedcontaminatedsupersededdisputed
Gemini 1.5 Flash (0514 preview)Google DeepMind77.8%not reported2024-05-14ind.saturatedcontaminatedsuperseded
Mixtral (8x22B)Mistral AI77.8%not reported2024-04-17ind.saturatedcontaminatedsuperseded
Phi-3-mediumMicrosoft77.5%not reported2024-04-23ind.saturatedcontaminatedsupersededdisputed
Amazon Nova LiteAmazon77.0%not reported2024-12-03ind.saturatedcontaminatedsuperseded
GPT-4o-miniOpenAI76.7%not reported2024-07-18ind.saturatedcontaminatedsupersededdisputed
Yi 34B01.AI76.3%not reported2023-11-02ind.saturatedcontaminatedsupersededdisputed
Yi 34B01.AI76.2%not reported2023-11-02ind.saturatedcontaminatedsupersededdisputed
Claude 3 SonnetAnthropic75.9%not reported2024-02-29ind.saturatedcontaminatedsuperseded
Gemma 2 Instruct (27B)Google DeepMind75.7%not reported2024-06-24ind.saturatedcontaminatedsuperseded
Phi-3-smallMicrosoft75.7%not reported2024-04-23ind.saturatedcontaminatedsuperseded
Qwen1.5 (32B)Alibaba74.4%not reported2024-02-04ind.saturatedcontaminatedsuperseded
Claude 3.5 HaikuAnthropic74.3%not reported2024-10-22ind.saturatedcontaminatedsuperseded
Gemini 1.5 Flash (002)Google DeepMind73.9%not reported2024-09-24ind.saturatedcontaminatedsuperseded
Claude 3 HaikuAnthropic73.8%not reported2024-03-07ind.saturatedcontaminatedsuperseded
Claude 2.1Anthropic73.5%not reported2023-11-21ind.saturatedcontaminatedsuperseded
Yi-Chat 34B01.AI73.5%not reported2023-11-22ind.saturatedcontaminatedsupersededdisputed
Qwen2.5 7BAlibaba72.9%not reported2024-09-19ind.saturatedcontaminatedsuperseded
Inflection-1Inflection AI72.7%not reported2023-06-22ind.saturatedcontaminatedsuperseded
Gemma 2 Instruct (9B)Google DeepMind72.1%not reported2024-06-24ind.saturatedcontaminatedsuperseded
GPT-3.5 version 1106OpenAI71.4%not reported2023-11-06ind.saturatedcontaminatedsuperseded
Amazon Nova MicroAmazon70.8%not reported2024-12-03ind.saturatedcontaminatedsuperseded
Falcon 180BTechnology Innovation Institute70.6%not reported2023-09-06ind.saturatedcontaminatedsuperseded
Mixtral 8x7BMistral AI70.6%not reported2023-12-11ind.saturatedcontaminatedsupersededdisputed
Mixtral 8x7BMistral AI70.5%not reported2023-12-11ind.saturatedcontaminatedsupersededdisputed
Gemini 1.0 ProGoogle DeepMind70.0%not reported2023-12-13ind.saturatedcontaminatedsuperseded
GPT-3.5OpenAI70.0%not reported2022-03-15ind.saturatedcontaminatedsuperseded
LLAMA 2 70BMeta AI69.9%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 70BMeta AI69.8%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 70BMeta AI69.5%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Command R+ (Aug '24)Cohere,Cohere Labs (formerly Cohere for AI)69.4%not reported2024-08-30ind.saturatedcontaminatedsuperseded
PaLM 540BGoogle Research69.3%not reported2022-04-04ind.saturatedcontaminatedsuperseded
GPT-3.5 TurboOpenAI68.9%not reported2023-06-13ind.saturatedcontaminatedsuperseded
LLAMA 2 70BMeta AI68.9%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Claude Instant 1.2Anthropic68.8%not reported2023-08-09ind.saturatedcontaminatedsuperseded
Mistral Large (2402)Mistral AI68.8%not reported2024-02-26ind.saturatedcontaminatedsuperseded
Phi-3-miniMicrosoft68.8%not reported2024-04-23ind.saturatedcontaminatedsuperseded
Qwen1.5 (14B)Alibaba68.6%not reported2024-02-04ind.saturatedcontaminatedsuperseded
StableBeluga2 70BStability AI68.6%not reported2023-07-20ind.saturatedcontaminatedsuperseded
Qwen2.5-Coder-7BAlibaba68.0%not reported2024-09-18ind.saturatedcontaminatedsuperseded
Yi-Chat 34B01.AI67.6%not reported2023-11-22ind.saturatedcontaminatedsupersededdisputed
Chinchilla 70BDeepMind67.5%not reported2022-03-29ind.saturatedcontaminatedsuperseded
GPT-3.5 TurboOpenAI67.3%not reported2024-01-25ind.saturatedcontaminatedsuperseded
Llama-3-In 8bMeta AI66.5%not reported2024-04-18ind.saturatedcontaminatedsuperseded
QWEN 14BAlibaba66.3%not reported2023-09-28ind.saturatedcontaminatedsupersededdisputed
Gemma 7BGoogle DeepMind66.1%not reported2024-02-21ind.saturatedcontaminatedsupersededdisputed
Qwen-Chat 14BAlibaba65.0%not reported2023-09-24ind.saturatedcontaminatedsupersededdisputed
Gemma 7BGoogle DeepMind64.3%not reported2024-02-21ind.saturatedcontaminatedsupersededdisputed
StarCoder2-15BHugging Face,ServiceNow,NVIDIA,BigCode64.1%not reported2024-02-20ind.saturatedcontaminatedsuperseded
Qwen-Chat 14BAlibaba64.0%not reported2023-09-24ind.saturatedcontaminatedsupersededdisputed
Yi 6B01.AI64.0%not reported2023-11-02ind.saturatedcontaminatedsupersededdisputed
Gemma 7BGoogle DeepMind63.6%not reported2024-02-21ind.saturatedcontaminatedsupersededdisputed
LLaMA 65BMeta AI63.4%not reported2023-02-24ind.saturatedcontaminatedsuperseded
Yi 6B01.AI63.2%not reported2023-11-02ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 34BMeta AI62.6%not reported2023-07-18ind.saturatedcontaminatedsuperseded
Qwen1.5 (7B)Alibaba62.6%not reported2024-02-04ind.saturatedcontaminatedsuperseded
Mistral 7BMistral AI62.5%not reported2023-12-11ind.saturatedcontaminatedsuperseded
Mistral 7BMistral AI62.5%not reported2023-09-27ind.saturatedcontaminatedsupersededdisputed
Mistral 7BMistral AI61.7%not reported2023-09-27ind.saturatedcontaminatedsupersededdisputed
Qwen2.5-Coder-32BAlibaba61.2%not reported2024-09-18ind.saturatedcontaminatedsupersededdisputed
Yi-Chat 6B01.AI61.0%not reported2023-11-22ind.saturatedcontaminatedsupersededdisputed
Mistral 7BMistral AI60.1%not reported2023-09-27ind.saturatedcontaminatedsupersededdisputed
GopherDeepMind60.0%not reported2021-12-08ind.saturatedcontaminatedsuperseded
LLaMA2-Chat 70BMeta AI59.9%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Mistral Instruct v0.3 (7B)Mistral AI59.9%not reported2024-05-27ind.saturatedcontaminatedsuperseded
LLaMA2-Chat 70BMeta AI59.4%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Baichuan2 13BBaichuan59.2%not reported2023-09-06ind.saturatedcontaminatedsupersededdisputed
Baichuan2 13BBaichuan59.2%not reported2023-09-06ind.saturatedcontaminatedsupersededdisputed
LLaMA 33BMeta AI58.7%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
Nemotron-4 15BNVIDIA58.7%not reported2024-02-26ind.saturatedcontaminatedsuperseded
Falcon2-11B stage 4Technology Innovation Institute58.4%not reported2024-05-09ind.saturatedcontaminatedsuperseded
Phi-2Microsoft58.4%not reported2023-12-12ind.saturatedcontaminatedsupersededdisputed
Yi-Chat 6B01.AI58.2%not reported2023-11-22ind.saturatedcontaminatedsupersededdisputed
LLaMA 33BMeta AI57.8%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
Falcon 40BTechnology Innovation Institute56.9%not reported2023-03-15ind.saturatedcontaminatedsupersededdisputed
LLaMA 33BMeta AI56.8%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
Mistral 7BMistral AI56.6%not reported2023-09-27ind.saturatedcontaminatedsupersededdisputed
Llama 3.2 Vision Instruct Turbo (11B)Meta AI56.5%not reported2024-09-24ind.saturatedcontaminatedsuperseded
Phi-2Microsoft56.3%not reported2023-12-12ind.saturatedcontaminatedsupersededdisputed
Llama 3.1 8BMeta AI56.1%not reported2024-07-23ind.saturatedcontaminatedsuperseded
LLaMA-2 13BMeta AI55.6%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Falcon 40BTechnology Innovation Institute55.4%not reported2023-03-15ind.saturatedcontaminatedsupersededdisputed
LLaMA-2 13BMeta AI55.4%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLaMA-2 13BMeta AI55.1%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLaMA-2 13BMeta AI54.8%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
Baichuan2 7BBaichuan54.2%not reported2023-09-20ind.saturatedcontaminatedsuperseded
Qwen2.5-Coder-1.5BAlibaba53.6%not reported2024-09-18ind.saturatedcontaminatedsuperseded
QWEN 14BAlibaba53.4%not reported2023-09-28ind.saturatedcontaminatedsupersededdisputed
Baichuan 1-13B-BaseBaichuan51.6%not reported2023-07-11ind.saturatedcontaminatedsuperseded
LLaMA2-Chat 13BMeta AI50.9%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI49.9%not reported2024-11-29ind.saturatedcontaminatedsuperseded
MPT 30BMosaicML47.9%not reported2023-06-22ind.saturatedcontaminatedsupersededdisputed
LLaMA 13BMeta AI47.7%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
LLaMA2-Chat 13BMeta AI47.3%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLaMA 13BMeta AI46.9%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
MPT 30BMosaicML46.9%not reported2023-06-22ind.saturatedcontaminatedsupersededdisputed
LLaMA 13BMeta AI46.3%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 7BMeta AI45.8%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 7BMeta AI45.7%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
LLAMA 2 7BMeta AI45.3%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
QWEN 7BAlibaba45.0%not reported2023-09-28ind.saturatedcontaminatedsuperseded
LLAMA 2 7BMeta AI44.4%not reported2023-07-18ind.saturatedcontaminatedsupersededdisputed
GPT-3OpenAI43.9%not reported2022-01-27ind.saturatedcontaminatedsuperseded
Baichuan 1-7BBaichuan42.3%not reported2023-06-01ind.saturatedcontaminatedsuperseded
Gemma 2BGoogle DeepMind42.3%not reported2024-02-21ind.saturatedcontaminatedsuperseded
DS-Coder-33B-BaseDeepSeek,Peking University39.4%not reported2023-11-02ind.saturatedcontaminatedsuperseded
StarCoder2-7BHugging Face,ServiceNow,NVIDIA,BigCode38.8%not reported2024-02-20ind.saturatedcontaminatedsuperseded
phi-1.5Microsoft37.6%not reported2023-09-11ind.saturatedcontaminatedsuperseded
StarCoder2-3BHugging Face,ServiceNow,NVIDIA,BigCode36.6%not reported2024-02-22ind.saturatedcontaminatedsuperseded
DS-Coder-6.7B-BaseDeepSeek,Peking University36.4%not reported2023-11-02ind.saturatedcontaminatedsuperseded
XGen-7BSalesforce36.3%not reported2023-06-27ind.saturatedcontaminatedsupersededdisputed
LLaMA 7BMeta AI35.6%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
LLaMA 7BMeta AI35.2%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
LLaMA 7BMeta AI35.1%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
Falcon 7BTechnology Innovation Institute35.0%not reported2023-04-24ind.saturatedcontaminatedsupersededdisputed
XGen-7BSalesforce32.1%not reported2023-06-27ind.saturatedcontaminatedsupersededdisputed
LLaMA 7BMeta AI32.0%not reported2023-02-24ind.saturatedcontaminatedsupersededdisputed
MPT 7BMosaicML30.8%not reported2023-05-05ind.saturatedcontaminatedsupersededdisputed
MPT 7BMosaicML27.9%not reported2023-05-05ind.saturatedcontaminatedsupersededdisputed
MPT 7BMosaicML27.4%not reported2023-05-05ind.saturatedcontaminatedsupersededdisputed
Falcon 7BTechnology Innovation Institute26.9%not reported2023-04-24ind.saturatedcontaminatedsupersededdisputed
MPT 7BMosaicML26.8%not reported2023-05-05ind.saturatedcontaminatedsupersededdisputed
MPT 7BMosaicML26.7%not reported2023-05-05ind.saturatedcontaminatedsupersededdisputed
Cerebras-GPT-13BCerebras Systems26.2%not reported2023-03-20ind.saturatedcontaminatedsupersededdisputed
Dolly-v2-12BDatabricks26.2%not reported2023-04-11ind.saturatedcontaminatedsupersededdisputed
Falcon 7BTechnology Innovation Institute26.2%not reported2023-04-24ind.saturatedcontaminatedsupersededdisputed
Falcon 7BTechnology Innovation Institute26.0%not reported2023-04-24ind.saturatedcontaminatedsupersededdisputed
DS-Coder-1.3B-BaseDeepSeek,Peking University25.8%not reported2023-11-02ind.saturatedcontaminatedsuperseded
GPT-J-6BEleutherAI,LAION25.7%not reported2021-08-05ind.saturatedcontaminatedsupersededdisputed
Dolly-v2-12BDatabricks25.4%not reported2023-04-11ind.saturatedcontaminatedsupersededdisputed
GPT-J-6BEleutherAI,LAION25.1%not reported2021-08-05ind.saturatedcontaminatedsupersededdisputed
Cerebras-GPT-13BCerebras Systems24.6%not reported2023-03-20ind.saturatedcontaminatedsupersededdisputed
Falcon 7BTechnology Innovation Institute23.9%not reported2023-04-24ind.saturatedcontaminatedsupersededdisputed

Vendor-reported

Reported by the model's own maker. No independent measurement of these numbers is on this site.

ModelScoreHarnessMeasuredProvenanceHealth
Claude 2Anthropic78.5%not reported2023-07-11vendorsaturatedcontaminatedvendor onlysuperseded
Claude 1.3Anthropic77.0%not reported2023-04-18vendorsaturatedcontaminatedvendor onlysuperseded
Claude Instant 1.1Anthropic73.4%not reportednot reportedvendorsaturatedcontaminatedvendor onlysuperseded
Claude Instant 1.2Anthropic73.2%not reported2023-08-09vendorsaturatedcontaminatedsuperseded

Trajectory

Best score measured on this site

0255075100GPT-J-6B: 25.7%GPT-J-6B: 25.1%Gopher: 60.0%GPT-3: 43.9%GPT-3.5: 70.0%Chinchilla 70B: 67.5%PaLM 540B: 69.3%LLaMA 65B: 63.4%LLaMA 33B: 58.7%LLaMA 33B: 57.8%LLaMA 33B: 56.8%LLaMA 13B: 47.7%LLaMA 13B: 46.9%LLaMA 13B: 46.3%LLaMA 7B: 35.6%LLaMA 7B: 35.2%LLaMA 7B: 35.1%LLaMA 7B: 32.0%GPT-4: 86.4%Falcon 40B: 56.9%Falcon 40B: 55.4%Cerebras-GPT-13B: 26.2%Cerebras-GPT-13B: 24.6%Dolly-v2-12B: 26.2%Dolly-v2-12B: 25.4%Falcon 7B: 35.0%Falcon 7B: 26.9%Falcon 7B: 26.2%Falcon 7B: 26.0%Falcon 7B: 23.9%MPT 7B: 30.8%MPT 7B: 27.9%MPT 7B: 27.4%MPT 7B: 26.8%MPT 7B: 26.7%Baichuan 1-7B: 42.3%GPT-4: 82.4%GPT-3.5 Turbo: 68.9%Inflection-1: 72.7%MPT 30B: 47.9%MPT 30B: 46.9%XGen-7B: 36.3%XGen-7B: 32.1%Baichuan 1-13B-Base: 51.6%LLAMA 2 70B: 69.9%LLAMA 2 70B: 69.8%LLAMA 2 70B: 69.5%LLAMA 2 70B: 68.9%LLAMA 2 34B: 62.6%LLaMA2-Chat 70B: 59.9%LLaMA2-Chat 70B: 59.4%LLaMA-2 13B: 55.6%LLaMA-2 13B: 55.4%LLaMA-2 13B: 55.1%LLaMA-2 13B: 54.8%LLaMA2-Chat 13B: 50.9%LLaMA2-Chat 13B: 47.3%LLAMA 2 7B: 45.8%LLAMA 2 7B: 45.7%LLAMA 2 7B: 45.3%LLAMA 2 7B: 44.4%StableBeluga2 70B: 68.6%Claude Instant 1.2: 68.8%Falcon 180B: 70.6%Baichuan2 13B: 59.2%Baichuan2 13B: 59.2%phi-1.5: 37.6%Baichuan2 7B: 54.2%Qwen-Chat 14B: 65.0%Qwen-Chat 14B: 64.0%Mistral 7B: 62.5%Mistral 7B: 61.7%Mistral 7B: 60.1%Mistral 7B: 56.6%QWEN 14B: 66.3%QWEN 14B: 53.4%QWEN 7B: 45.0%Yi 34B: 76.3%Yi 34B: 76.2%Yi 6B: 64.0%Yi 6B: 63.2%DS-Coder-33B-Base: 39.4%DS-Coder-6.7B-Base: 36.4%DS-Coder-1.3B-Base: 25.8%GPT-4 Turbo: 79.6%GPT-3.5 version 1106: 71.4%Claude 2.1: 73.5%Yi-Chat 34B: 73.5%Yi-Chat 34B: 67.6%Yi-Chat 6B: 61.0%Yi-Chat 6B: 58.2%Mixtral 8x7B: 70.6%Mixtral 8x7B: 70.5%Mistral 7B: 62.5%Phi-2: 58.4%Phi-2: 56.3%Gemini 1.0 Pro: 70.0%GPT-3.5 Turbo: 67.3%Qwen1.5 (32B): 74.4%Qwen1.5 (14B): 68.6%Qwen1.5 (7B): 62.6%Gemini 1.5 Pro (Feb 2024): 81.9%StarCoder2-15B: 64.1%StarCoder2-7B: 38.8%Gemma 7B: 66.1%Gemma 7B: 64.3%Gemma 7B: 63.6%Gemma 2B: 42.3%StarCoder2-3B: 36.6%Mistral Large (2402): 68.8%Nemotron-4 15B: 58.7%Claude 3 Opus: 84.6%Claude 3 Sonnet: 75.9%Claude 3 Haiku: 73.8%GPT-4 Turbo (2024-04-09): 81.3%Mixtral (8x22B): 77.8%Llama-3-In 8b: 66.5%Phi-3-medium: 78.0%Phi-3-medium: 77.9%Phi-3-medium: 77.5%Phi-3-small: 75.7%Phi-3-mini: 68.8%DeepSeek-V2 Base: 78.4%Falcon2-11B stage 4: 58.4%GPT-4o: 84.2%Yi Large (Preview): 79.3%Gemini 1.5 Pro (May 2024): 85.9%Gemini 1.5 Flash (0514 preview): 77.8%Gemini 1.5 Flash (001): 77.9%Mistral Instruct v0.3 (7B): 59.9%OpenHands + Qwen-2-72b: 82.4%Claude 3.5 Sonnet (20240620): 86.5%Gemma 2 Instruct (27B): 75.7%Gemma 2 Instruct (9B): 72.1%GPT-4o-mini: 81.8%GPT-4o-mini: 76.7%LLaMA 3 405B: 84.5%LLaMA-3.1 405B Base: 84.4%Llama 3.1 70B: 80.1%Llama 3.1 8B: 56.1%Mistral Large 2 (2407): 80.0%GPT-4o (2024-08-06): 84.3%Command R+ (Aug '24): 69.4%Qwen2.5-Coder-32B: 79.1%Qwen2.5-Coder-7B: 68.0%Qwen2.5-Coder-32B: 61.2%Qwen2.5-Coder-1.5B: 53.6%Qwen2.5 72B: 85.3%Qwen2.5 72B Base: 85.0%Qwen2.5 72B: 83.4%Qwen2.5 14b instruct: 79.9%Qwen2.5 7B: 72.9%Gemini 1.5 Pro (002): 86.9%Llama 3.2 Vision Instruct Turbo (90B): 80.3%Gemini 1.5 Flash (002): 73.9%Llama 3.2 Vision Instruct Turbo (11B): 56.5%Claude 3.5 Sonnet (20241022): 87.3%Claude 3.5 Haiku: 74.3%INTELLECT-1-Instruct: 49.9%Amazon Nova Pro: 82.0%Amazon Nova Lite: 77.0%Amazon Nova Micro: 70.8%Llama 3.3 70B: 86.3%Llama 3.3 70B: 79.1%Gemini 2.0 Flash (Experimental): 79.7%Phi-4 14b: 84.8%DeepSeek-V3: 87.2%DeepSeek-V3: 87.1%GPT-J-6B: 25.7%, best at the timeGopher: 60.0%, best at the timeGPT-3.5: 70.0%, best at the timeGPT-4: 86.4%, best at the timeClaude 3.5 Sonnet (20240620): 86.5%, best at the timeGemini 1.5 Pro (002): 86.9%, best at the timeClaude 3.5 Sonnet (20241022): 87.3%, best at the time20212024
Each dot is a score measured on this site; the line steps up at each new best and stays flat until the next one. A line that has stopped climbing is what saturation looks like.

As benchwiki records it

human baselineGPT-3 175B — 43.9 on 2020-09-07 (independent)Gopher 280B — 60 on 2021-12-08 (vendor-reported)Chinchilla 70B — 67.6 on 2022-03-29 (vendor-reported)PaLM 540B — 69.3 on 2022-04-05 (vendor-reported)GPT-4 — 86.4 on 2023-03-14 (vendor-reported)Claude 3 Opus — 86.8 on 2024-03-04 (vendor-reported)GPT-4o — 88.7 on 2024-05-13 (vendor-reported)Llama 3.1 405B — 88.6 on 2024-07-23 (vendor-reported)20202024900
independent · vendor-reported. Points are not joined: the scores come from different models, harnesses and protocol versions, and a line between them would claim a continuity the data does not have.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

Source

This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2026-07-21; fetched 2026-10-19 by this site.