Needle-in-a-Haystack

What it measures

Synthetic retrieval stress test that hides facts at different depths and context lengths, then asks the model to recover them.

Primary metric

retrieval accuracy over context length by depth cells

Judge model

not judged by a model

Human baseline

—not reportedNo protocol-comparable human baseline is recorded; the base task is meant as a machine retrieval stress test.

Languages

English

Health

Report the full context-length by depth grid. A single average hides lost-in-the-middle failures.

Status evidence

Vanilla single-needle retrieval is now a smoke test rather than a discriminative frontier benchmark; RULER explicitly notes that models can be nearly perfect on vanilla NIAH while failing harder long-context tasks.

Saturated since

2024-04-09

Refresh cycle

continuous

Test set public

yes

Needles can be generated per run, but default haystacks, prompts, and configs are public; reuse of fixed seeds weakens the test.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.