MMLU-Redux

What it measures

A manual error audit of MMLU: experts re-annotated thousands of questions and found ~6.5% are broken — the evidence base for MMLU's label-noise ceiling.

Primary metric

accuracy on the re-annotated (error-corrected) MMLU questions (%)

Judge model

not judged by a model

Human baseline

—not reportedNo separate human baseline. MMLU-Redux inherits MMLU's context, where an ~89.8% expert baseline was the historical reference. Redux's contribution is orthogonal to a human score: it measures how much of MMLU's headroom was illusory (label noise), not human vs model performance. See the parent MMLU page for the baseline discussion.

Languages

English

Health

The original audit re-annotated 3,000 questions across 30 subjects (14 expert annotators); the expanded Redux 2.0 covers 5,700 across all 57 subjects. Overall ~6.5% of questions contain errors, but the rate is wildly uneven — near zero in some subjects, and as high as ~57% in Virology (the original 30-subject audit already flagged it above 25%). This unevenness is the point: a benchmark's effective ceiling is set by its worst subjects, so per-subject error rates matter more than the headline average.

Status evidence

As a competition benchmark, MMLU-Redux is a corrected slice of MMLU: it inherits MMLU's contamination and frontier convergence, so it does not differentiate top models — saturated by CLAUDE.md's definition (usefulness as a model-differentiating eval). Its ENDURING value is different and non-competitive: it remains an actively-used forensic reference and audit methodology for measuring label noise. Status here reflects the eval role; the still-active reference role is documented in prose. saturated_date is set to its launch — it described an already-saturated parent from birth.

Saturated since

2024-06-06

Refresh cycle

static

Test set public

yes

Inherits MMLU's contamination completely — these ARE MMLU questions, corrected. Public since 2020 (as MMLU), fully static, ungated. Correcting the labels does nothing to reduce memorization risk; if anything the questions are more discussed now. Risk is high for the same reasons as the parent.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.