QASPER

What it measures

Scientific-paper QA over full NLP papers, with abstractive, extractive, yes/no, and unanswerable answers plus supporting evidence.

Primary metric

answer F1 and evidence F1

Judge model

not judged by a model

Human baseline

—not reportedThe paper reports that contemporary models lagged humans by at least 27 F1 points, but no single site-compatible aggregate human baseline is recorded here.

Languages

English

Health

The full dataset has 5,049 questions over 1,585 NLP papers; document length and multiple answer types make a single binomial confidence interval misleading.

Status evidence

QASPER remains useful as a diagnostic component, but as a 2021 public static standalone benchmark it is mostly consumed through SCROLLS and LongBench rather than trusted as a current frontier long-context leaderboard.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The data is public and static. Because source papers, questions, and answers are available on Hugging Face, post-release training contamination is plausible.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• AllenAI QASPER dataset cardofficial