FrontierMath

What it measures

Original, unpublished research-level math problems by expert mathematicians — mostly private to resist contamination. Frontier models solved under 2% at launch.

Primary metric

accuracy — % of problems solved, automated answer verification (%)

Judge model

not judged by a model

Human baseline

—not reportedNo aggregate human baseline exists, and — as with HLE — none is well-defined in principle, only more so. These are research-level problems spanning many advanced fields; no single mathematician can solve across all of them. The meaningful human reference is per-problem: an expert in the relevant subfield solves it over hours to days. Fields Medalists who reviewed the problems underscored the point — Terence Tao said that in the near term 'the only way to solve them, short of having a real domain expert in the area, is to get a lucky overfit,' and predicted the hardest tier would 'resist AIs for several years.'

Languages

English

Health

Scores are reported per tier, and the tiers are small (Tier 4 is 43 problems), so per-tier figures carry wide confidence intervals — a few problems move the percentage several points. Two deeper caveats dominate any FrontierMath number: (1) tools-on vs tools-off and heavy-compute vs single-pass swing results enormously (see Scoring methodology); (2) the set is private, so you are trusting Epoch's evaluation rather than reproducing it yourself. Always cite the tier, the tool/compute setting, and that it is an Epoch-run private-set number.

Status evidence

The strongest active case on the wiki. At launch (Nov 2024) state-of-the-art models solved UNDER 2% — a benchmark frontier systems almost entirely fail. Reasoning models with Python tools have since climbed (OpenAI announced o3 at 25.2% in Dec 2024, tools-on), but that figure is contested — Epoch's own independent reproductions of comparable models came in far lower (~11%). By the June 2026 v2 release Epoch added a harder Tier 4 precisely because Tiers 1–3 progress had become significant, yet the hardest tier remains far from solved. Current figures move fast and are tool/compute-dependent — consult Epoch's live dashboard; this page does not restate unverified third-party numbers. Saturation here would mean research-level, closed-form mathematics is largely automated.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

The strongest contamination defense catalogued on the wiki — and the counter-model to AIME's yearly refresh and MMLU's fully-public exposure. The problems are ORIGINAL and UNPUBLISHED, and the benchmark is mostly PRIVATE by design: only a small sample set is public, and Epoch runs evaluations on a held-out set that is not released. The tradeoff is a different threat model: you cannot independently reproduce a private benchmark, so you must trust the evaluator (see Known issues). The privacy is also entangled with the funding history — OpenAI, which funded the dataset, had access to most problems, so Epoch established held-out sets the funder cannot see (53 solutions withheld in one version; 20 of 50 Tier 4 problems reserved) as a post-hoc safeguard.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Epoch AI (official / canonical tracker)official