MATH

What it measures

12,500 AMC/AIME-level competition problems with worked solutions — usually evaluated on its 500-problem MATH-500 subset, the reasoning-model math standard.

Primary metric

accuracy on MATH-500 — math-equivalence match on the boxed answer (%)

Judge model

not judged by a model

Human baseline

90The paper reports a stark human RANGE, not a point: a three-time IMO gold medalist scored 90% while a computer-science PhD student who dislikes math scored ~40%. The gauge uses the IMO-gold 90% as the reference line — the expert ceiling that reasoning models have now surpassed — and the 40% lower bound lives here. For competition math, 'human performance' is nearly meaningless without naming the human; this is the starkest range-not-a-point case on the wiki (cf. AGIEval, GPQA).

Languages

English

Health

MATH-500 is 500 problems → ±~2.5pt (95% CI) near 90%. The 7 subjects × 5 difficulty levels enable fine-grained analysis, but any single level×subject cell is tiny (tens of problems) — never compare models on 'Level-5 Geometry' alone. As at the saturated frontier, aggregate gaps under a few points are noise; the bigger issue is equivalence-checker disagreement (see Scoring methodology).

Status evidence

MATH resisted far longer than GSM8K: GPT-4 managed only ~42% on the full set in 2023. But the reasoning-model era ended it — o1 (2024) reached ~94.8% on MATH-500 and later models sit in the 90s, so it no longer discriminates at the frontier. saturated_date 2024-09 marks the o1 inflection. As with GSM8K, saturated ≠ solved: exact-answer scoring hides whether the reasoning was valid, and headroom has moved to AIME and FrontierMath.

Saturated since

2024-09-01

Refresh cycle

static

Test set public

yes

None, and compounded. Public since 2021, MIT-licensed, with full worked solutions mirrored across the web. Worse: the PRM800K release (2023) put step-level solutions to MATH TRAINING problems into wide circulation as process-supervision data — valuable for research, but more of the benchmark's solution structure in the open. MATH-500's cost motivation is partly entangled with this: it is drawn to avoid the train/PRM overlap and to make grading tractable. Fully static, no refresh. Treat near-ceiling scores as memorization-inclusive.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Papers with Code (MATH)third-party