GPQA

What it measures

PhD-written science questions so hard that skilled non-experts with Google score 34% — the 'Google-proof' exam, reported on its 198-question Diamond subset.

Primary metric

accuracy on GPQA Diamond, 4-option multiple choice (%)

Judge model

not judged by a model

Human baseline

81.3PhD-domain experts scored 81.3% on the Diamond subset (the primary configuration this page reports); non-expert validators with unrestricted web access scored only 22.1% on Diamond. On the full 448-question main set the headline figures are ~65% expert (74% after discounting mistakes the experts themselves flagged in retrospect) and ~34% non-expert. IMPORTANT nuance: OpenAI, evaluating o1, recruited its own PhD experts who scored only 69.7% on Diamond — so whether a model 'beats PhD experts' depends entirely on which expert pool you compare against. The gauge/timeline use the paper's 81.3% Diamond expert accuracy as the reference line, because GPQA's entire claim is expert-level difficulty; beating the 22–34% non-expert line is trivial and not the point.

Languages

English

Health

GPQA Diamond is only 198 questions. At ~80% accuracy that is a 95% confidence interval of roughly ±5.6 points (±7 near 50%). This is the single most important number on the page: a great many publicized 'Model A beats Model B by 2 points on GPQA Diamond' claims are statistically indistinguishable from noise. Treat any Diamond gap under ~6 points as unresolved, and prefer the larger main/extended sets when a comparison actually needs to be settled.

Status evidence

Frontier reasoning models reach the high 80s on GPQA Diamond (e.g. Grok 4 ~87%, mid-2025), against a label-noise-adjusted ceiling likely in the low 90s — the paper documents a few percent of questions with expert-identified errors. Unlike MMLU, meaningful headroom remains and top models are only now crossing the PhD-expert line, so it still discriminates; but the trajectory is one or two model generations from done. Judgment call — the first `nearing-saturation` benchmark on the wiki.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The dataset is public but gated: HuggingFace access requires accepting an agreement, and the authors ask users NOT to post questions or answers online 'to reduce the risk of leakage into foundation model training corpora.' This is a voluntary pledge, not a technical control — the same class of opt-in decontamination that failed for BIG-Bench's canary. The set is fully static with no refreshed variant, and Diamond questions have been discussed and reproduced online since 2023, which pushes the effective risk from medium toward high over time.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Epoch AI (GPQA Diamond)third-party
• Artificial Analysis (GPQA Diamond)third-party