MMLU-Pro

What it measures

MMLU rebuilt to fix its flaws: 10 options instead of 4, trivia and label noise filtered out, and reasoning-heavy questions that reward chain-of-thought.

Primary metric

accuracy, 10-option multiple choice (%)

Judge model

not judged by a model

Human baseline

—not reportedNo measured human baseline. MMLU-Pro's difficulty was calibrated by model-side filtering (removing questions strong models found trivial and adding reasoning-heavy ones), not by testing humans — so unlike its parent MMLU (famous for its 89.8% expert baseline framing) and unlike GPQA (built around expert vs non-expert gaps), the successor here has no human reference line. A notable irony worth flagging when reading 'above/below human' claims: there is no human number to compare against.

Languages

English

Health

12,032 questions give tight aggregate CIs (±~0.6pt at 85%), and the 10-option format lowers the random-guess floor from 25% to 10%, so scores reflect knowledge more than test-taking. But per-category splits (14 disciplines) are much smaller — do not compare models on a single discipline. As frontier scores compress into the high 80s, top-model gaps under a couple of points are again approaching noise.

Status evidence

Frontier models cluster in the high 80s on MMLU-Pro by 2025–26 (leaderboards put the top ~89–90%), but cleaner labels and 10-option questions push the effective ceiling above MMLU's ~90%, so it still separates strong models where MMLU cannot. The 16–33 point accuracy drop it induced over MMLU is eroding; the trajectory points at saturation within a generation or two, hence nearing-saturation rather than active.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Public and ungated on HuggingFace, and fully static with no refreshed variant. Contamination risk is lower than MMLU's mainly because it launched in mid-2024 — less crawl exposure so far — but it is rising, and many questions are drawn from public STEM problem sets (e.g. exam and textbook sources) that may themselves appear in training data. Treat it as medium now, trending up.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• MMLU-Pro (official, HuggingFace)official
• Artificial Analysis (MMLU-Pro)third-party