RULER

What it measures

Configurable synthetic benchmark that expands needle retrieval into multi-needle, multi-hop tracing, and aggregation tasks.

Primary metric

accuracy across configured synthetic tasks

Judge model

not judged by a model

Human baseline

—not reportedNo protocol-comparable human baseline is recorded; the tasks are synthetic model diagnostics.

Languages

English

Health

Report score by context length and task type. The visual graph on this page deliberately plots only the official repository's Avg. column: mean RULER accuracy across the 4K, 8K, 16K, 32K, 64K, and 128K columns for selected high-scoring models in the 2026-08-12 README snapshot. It does not plot claimed length, effective length, weighted averages, or per-length degradation.

Status evidence

RULER is still a current synthetic stress test, but newer top rows report mid-90s average scores across 4k-128k, so it is approaching ceiling for the strongest long-context models while still exposing degradation.

Saturated since

not saturated

Refresh cycle

continuous

Test set public

yes

RULER can generate configurable synthetic instances; using fresh seeds and held-out configs reduces memorization risk.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official RULER repositoryofficial