SimpleQA

What it measures

4,326 short fact-seeking questions chosen to stump GPT-4 — a hallucination test graded correct / incorrect / not-attempted.

Primary metric

correct (accuracy over all questions, %)

Judge model

Prompted ChatGPT classifier that compares the predicted answer to the reference (GPT-4o in OpenAI's reference implementation, simple-evals). Scores graded by different grader models or prompt versions are not strictly comparable.

Human baseline

—not reportedNo human baseline — and arguably none is meaningful. The questions were deliberately selected to be obscure (adversarially filtered so frontier models got them wrong), so a human sitting the test closed-book would mostly measure luck, and open-book would measure Googling, not knowledge. The paper reports a 94.4% agreement rate from a third annotator, but that measures dataset quality (are the answers indisputable?), not human question-answering ability. This is the first benchmark on the wiki where a human reference line is not just missing but conceptually beside the point.

Languages

English

Health

4,326 questions give a reasonably tight aggregate CI (±~1.5pt at 40% correct). The more important caveats are non-sampling: the LLM grader introduces its own error and version drift (see Known issues), and because SimpleQA reports three outcomes, a single 'accuracy' number hides whether a model is knowledgeable or merely bold — always read 'correct' alongside 'not attempted'.

Status evidence

Still discriminating hard. Frontier models remain well short of the ceiling on parametric recall: GPT-4o 38% correct at launch, GPT-4.5 ~62% by early 2025 — climbing but nowhere near solved, and reasoning-tuned models do not automatically help (o1-mini scored 8%). What would move it off 'active': frontier 'correct' crossing ~85–90% (approaching the label ceiling), OR retrieval-augmented deployment becoming universal enough that closed-book parametric recall stops mattering. Neither has happened.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Public and ungated (openai/simple-evals), fully static, no refresh cycle. The irony is acute: a benchmark built to measure hallucination can itself be memorized. It was adversarially collected against 2024-era models, so the exact Q&A pairs — now discussed in papers, blog posts, and leaderboards — increasingly enter post-2024 training corpora, and a model can 'improve' by having seen the answer key rather than by knowing more. SimpleQA Verified (2025) partly re-collects and cleans to push back on this.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• SimpleQA (official, OpenAI)official
• Artificial Analysis / third-party trackersthird-party