Humanity's Last Exam

What it measures

~2,500 expert-written questions across 100+ subjects, each filtered to stump frontier models — designed to be the final closed-ended academic benchmark.

Primary metric

accuracy, no tools (%)

Judge model

GPT-4o (gpt-4o-2024-08-06, structured decoding) checks whether a model's answer is equivalent to the reference for the 80% short-answer questions. Judge model/version drift makes scores across grader versions not strictly comparable — pin the grader.

Human baseline

—not reportedNo human baseline — and here the concept itself breaks down. HLE spans 100+ expert subjects from classics to rocket engineering; no single human could sit the whole exam, so an aggregate 'human score' is undefined in principle, not merely unmeasured. This differs from SimpleQA, where a human COULD take the test but the score would mostly measure Googling: HLE is superhuman in BREADTH by construction. The only meaningful human reference is per-question — each was solved by the domain expert who wrote it — which is exactly why the benchmark exists: to test whether one model can match the union of many human specialists.

Languages

English

Health

~2,500 questions give a workable aggregate CI (±~2pt at 40%), but two non-sampling caveats dominate. First, scores move FAST — a published number is often stale within months (single digits to ~38% in under a year), so always read HLE figures with their date and this page's last_updated. Second, the number is meaningless without its CONFIGURATION: no-tools vs tool-augmented vs multi-agent 'heavy' modes differ by tens of points on the same model (see Timeline). Read accuracy beside calibration error, and never compare across tool settings.

Status evidence

The clearest active benchmark on the wiki. Launch scores were single digits (GPT-4o 3.3%, o1 9.1%, DeepSeek-R1 9.4%, no tools); the current no-tools frontier is ~38% (Gemini 3 Pro, Nov 2025) — a steep climb but a long way from solved. Given the 'last exam' thesis, saturation here would mean closed-ended academic testing has run out of headroom entirely, so 'what replaces it' points toward open-ended, agentic, and interactive evaluation rather than a harder Q&A set. Note: tool-augmented and agent scores run much higher and are tracked separately (see Timeline).

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The strongest mitigation design catalogued on the wiki so far: alongside the public ~2,500-question set, CAIS/Scale hold a PRIVATE set of questions used to detect benchmark-specific overfitting by comparing a model's public vs private performance — a static-but-audited design. The public set is gated on HuggingFace (accept an agreement; users are asked not to re-share or re-upload). This does not make it leak-proof — the public questions are discussed online and the failure-filter selected for 2024-frontier blind spots — but the private audit catches the crudest gaming, which is why the risk is medium rather than high despite a fully static public set.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Humanity's Last Exam (official, CAIS)official
• Scale AI / SEALthird-party