TruthfulQA

What it measures

817 adversarial questions probing whether models repeat common human misconceptions — famous for finding that bigger models were often less truthful.

Primary metric

MC2 (normalized probability mass on true answers, %)

Judge model

not judged by a model

Human baseline

9494% truthful on the generation task in the original paper (human validators). No published human baseline exists for the MC1/MC2 multiple-choice variants, so the timeline and gauge track generation truthfulness to stay comparable to this number.

Languages

English

Health

817 questions → roughly ±3pt across the 60–90% range (95% CI). The 38 category splits (some <20 questions) are far too small for per-category comparison. Metric choice (MC1 vs MC2 vs generation) moves scores more than most model-to-model gaps do.

Status evidence

By 2023 post-RLHF models exceeded the 94% human truthfulness baseline — but largely because the exact misconceptions TruthfulQA targets are what alignment tuning trains against, and the dataset itself leaked into alignment mixes. Saturation here reflects targeted contamination, not broadly truthful models. saturated_date (2023-06) marks the ChatGPT/GPT-4 era when the inverse-scaling trend reversed.

Saturated since

2023-06-01

Refresh cycle

static

Test set public

yes

None, and uniquely self-defeating: the misconceptions TruthfulQA targets are exactly what RLHF and Constitutional-style alignment tuning is trained to avoid, and the dataset itself appears in some alignment/instruction mixes. Modern scores partly measure exposure to the benchmark's own patterns.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Open LLM Leaderboard (v1, archived)third-party