SuperGLUE

What it measures

Eight English NLU tasks spanning question answering, inference, causal reasoning, word sense, and coreference.

Primary metric

SuperGLUE aggregate score (%)

Judge model

not judged by a model

Human baseline

89.8Official aggregate estimate. Per-task estimates mix prior published results with crowdworker reannotation; paired metrics are averaged before the eight-task macro-average.

Languages

English

Health

The aggregate is not a sample-size-weighted accuracy and has no single binomial confidence interval. CB, COPA, and WSC have especially small evaluation splits, while metric and prompt implementation differences can exceed small leaderboard gaps.

Status evidence

The official aggregate human estimate (89.8) was crossed by a single DeBERTa model (89.9) by 2020-12-29; the 2022 leaderboard leader reached 91.3, leaving a narrow, noisy ceiling.

Saturated since

2020-12-29

Refresh cycle

static

Test set public

yes

Official test labels were kept private behind the evaluation server, but test inputs, training and development data, source corpora, and benchmark examples are static and public. Modern harnesses normally score the labeled public validation splits, so those runs have no held-out-label protection.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official SuperGLUE leaderboardofficial