GSM-Hard

What it measures

GSM8K test problems with unusually large numerical values, designed to separate reasoning from easy arithmetic.

Primary metric

final-answer accuracy (%)

Judge model

not judged by a model

Human baseline

—not reportedNo aggregate human baseline was published for the automatically transformed set.

Languages

English

Health

At 1,319 items, a 50% accuracy has an approximate 95% binomial interval of +/-2.7 points, but malformed transformations introduce non-sampling error that a confidence interval cannot capture.

Status evidence

The public 1,319-item transformation remains a useful arithmetic stress test, but automatic number replacement creates implausible and sometimes invalid questions; it should not be used as a standalone reasoning benchmark.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

None beyond replacing GSM8K numbers; questions, generated programs, and targets are public.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.