LongBench-Pro

What it measures

Realistic bilingual long-context benchmark with 1,500 natural samples across 11 primary and 25 secondary tasks from 8k to 256k tokens.

Primary metric

normalized task-specific aggregate

Judge model

not judged by a model

Human baseline

—not reportedThe construction pipeline includes expert validation, but no single protocol-comparable human score is recorded here.

Languages

Chinese, English

Health

The benchmark is intentionally stratified by context requirement, length, and difficulty; aggregate scores should be accompanied by those slices. The visual graph on this page deliberately plots only the official LongBench-Pro paper Overall column for the supplied top-10 ranked models. It does not plot per-task, per-language, per-length, or difficulty-sliced scores; reasoning/thinking status is shown in the table text, not encoded as a separate plotted metric.

Status evidence

Newer than LongBench v1/v2 and designed for more realistic, bilingual, fine-grained long-context analysis across 8k-256k token inputs.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The released test file is public on Hugging Face; the benchmark relies on natural documents and expert validation, not on secrecy.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official LongBench-Pro repositoryofficial
• LongBench-Pro dataset cardofficial