BIG-Bench

What it measures

204-task community-built mega-suite spanning reasoning, knowledge, math, code and social bias — the sprawling proving ground BBH was distilled from.

Primary metric

normalized preferred-metric aggregate (0–100)

Judge model

not judged by a model

Human baseline

—not reportedThe paper collected per-task human-rater performance (average rater and best rater), but reports no single suite-wide aggregate human score we can cite here — human performance is only meaningful task by task.

Languages

English, Multilingual

Health

A suite-level number hides enormous per-task variance: tasks range from a handful to thousands of examples, and from trivial to near-impossible. Never compare models on the BIG-Bench aggregate alone — the task mix and weighting dominate the headline figure.

Status evidence

No longer maintained as an evaluation target. Its 'canary' GUID — meant to keep tasks out of training data — was widely ignored, so the suite leaked into web-scale corpora; task quality is famously uneven; and the field moved to distillates like BBH. Deprecated rather than saturated: models don't uniformly max it, it simply stopped being used.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Every task file embeds a 'canary' GUID string that trainers are asked not to strip, so they can detect and exclude BIG-bench from crawled data. In practice the canary was widely ignored, and the suite leaked into training corpora anyway — the field's most-cited example of a decontamination mechanism that depends on voluntary compliance and therefore failed.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.