ZeroSCROLLS

What it measures

Zero-shot long-text suite derived from SCROLLS with test-only tasks and new aggregation-style long-context challenges.

Primary metric

task-normalized zero-shot aggregate

Judge model

not judged by a model

Human baseline

—not reportedNo single suite-wide human baseline is recorded here; component tasks have their own annotation histories.

Languages

English

Health

The validation sets are tiny and meant for eyeballing; use held-out leaderboard evaluation for claims. The visual graph on this page deliberately plots only the official ZeroSCROLLS paper Table 3 Avg column from the leaderboard at the time of writing. It does not plot per-task scores or later leaderboard submissions.

Status evidence

ZeroSCROLLS fixed SCROLLS' fine-tuning mismatch and has a live leaderboard, but its 2023-era tasks are now shorter and narrower than current 100k+ and multi-needle long-context stress suites.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

ZeroSCROLLS removes training data and uses small validation sets, but the task definitions and component sources are public.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• ZeroSCROLLS dataset cardofficial