QuALITY

What it measures

Multiple-choice long-document QA where writers and validators read the full article, with a hard subset designed to defeat skimming.

Primary metric

multiple-choice accuracy (%)

Judge model

not judged by a model

Human baseline

93.5The NAACL paper reports 93.5% human performance and a 55.4% baseline-model score in the abstract.

Languages

English

Health

Report all-question and hard-subset accuracy separately; the hard subset was defined by speed-validation failures, not by model performance.

Status evidence

QuALITY still tests long-document multiple-choice comprehension, but its 2021-era context lengths are short for modern long-context models and the static data is better treated as a component diagnostic.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

The repository says test annotations are withheld for leaderboard submissions, but train/dev data and article sources are public.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official QuALITY leaderboardofficial