MGSM

What it measures

A human-translated 250-problem GSM8K subset for comparing grade-school mathematical reasoning across ten languages.

Primary metric

macro-average exact-answer accuracy across languages (%)

Judge model

not judged by a model

Human baseline

—not reportedNo single aggregate human baseline was published; the arithmetic is grade-school level, while translation and language competence are part of the task.

Languages

Bengali, Chinese, French, German, Japanese, Russian, Spanish, Swahili, Telugu, Thai

Health

Each language has only 250 paired items: near 50% accuracy the per-language 95% binomial interval is about +/-6.2 points. Cross-language errors are correlated because every language translates the same underlying problems.

Status evidence

The small, static translations are broadly supported by modern multilingual models, while large language gaps and prompt sensitivity still make MGSM useful diagnostically; newer perturbation-based successors test robustness more directly.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Human translation reduces exact English-string overlap but the underlying GSM8K questions, all translations, targets, and exemplars are public.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory