BIG-Bench Hard

What it measures

23 BIG-Bench tasks where models trailed humans — the benchmark that proved chain-of-thought prompting works, then fell to the reasoning-model era.

Primary metric

exact-match accuracy, averaged over 23 tasks (%)

Judge model

not judged by a model

Human baseline

67.7Average human-rater performance across the 23 tasks in the original paper; the best (max) human rater averaged ~94.4%. BBH tasks were selected precisely as the BIG-Bench tasks where prior models fell below the 67.7% average-rater line. (Canonical Table 1 values from the paper.)

Languages

English

Health

6,511 examples across 23 unequally-sized tasks; because the aggregate is an unweighted task average, a small task moves the number as much as a large one. Now that the mean is saturated, per-task scores — not the 23-task average — are where any remaining signal lives.

Status evidence

Frontier and reasoning-tuned models exceed ~90% on the 23-task average and are near-perfect on many individual tasks; the reasoning-model era made BBH trivial. Google DeepMind built BBEH (2025) — replacing each task with a harder counterpart — precisely because BBH stopped discriminating at the top.

Saturated since

2024-06-01

Refresh cycle

static

Test set public

yes

Inherits BIG-bench's public tasks and its widely-ignored canary GUID. BBH's popularity made it especially likely to appear in pretraining and instruction-tuning data, and there is no refreshed variant. Its high contamination risk is a direct reason BBEH was built with fresh tasks.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• llm-stats (BBH)third-party