GSM8K

What it measures

8.5K grade-school math word problems (2–8 arithmetic steps) — the default elementary-math benchmark and the classic chain-of-thought demonstration.

Primary metric

accuracy — exact match on the final numeric answer (%)

Judge model

not judged by a model

Human baseline

—not reportedNo measured human baseline. The paper states only an informal difficulty target: 'a bright middle school student should be able to solve every problem.' Because the problems are grade-school and were verified solvable, the implicit human ceiling is near 100% for a capable solver — but no human solve-rate was measured, so the score is null. Treat this like MMLU's estimated-baseline situation, only softer: a target description, not a number.

Languages

English

Health

1,319 test problems → ±~1.2pt (95% CI) near 95%, so at the saturated frontier ordinary model gaps are within noise. But the dominant caveat here is NOT sampling error — it is robustness: GSM-Symbolic shows the same model's accuracy swings by tens of points under trivial rewordings, so a single GSM8K number overstates reasoning reliability far more than its CI suggests. Also non-comparable across protocols (direct vs CoT vs maj@k).

Status evidence

Frontier models cleared ~95% by 2023–24 and the reasoning-model era pushed scores to ~97–99%, collapsing headroom. But saturation here is the sharpest 'saturated ≠ solved' case on the wiki: GSM-Symbolic (Apple, 2024) showed that changing only names/numbers, and especially adding one irrelevant clause (GSM-NoOp), drops accuracy heavily across models — so the SCORE is saturated while the CAPABILITY (robust arithmetic reasoning) is not. saturated_date 2024-01 marks the point frontier scores stopped discriminating; the discriminating signal moved to harder math (MATH → AIME → FrontierMath) and to robustness variants.

Saturated since

2024-01-01

Refresh cycle

static

Test set public

yes

None. Public since 2021, MIT-licensed, and mirrored across the web WITH worked natural-language solutions — ideal memorization fodder. Fully static, no refresh. Structurally the same exposure that undid MMLU, and a direct reason the GSM-Symbolic and GSM8K-Platinum variants exist: to test whether a score survives when the exact public items are perturbed or cleaned.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Papers with Code (GSM8K)third-party