GLUE

What it measures

Nine-task English NLU suite combining acceptability, sentiment, similarity, paraphrase, inference and coreference into one score.

Primary metric

GLUE Score (0-100)

Judge model

not judged by a model

Human baseline

87.1Conservative non-expert crowd estimate from Nangia and Bowman (2019), based on brief instructions and 20 demonstrations per task; it is not an expert ceiling.

Languages

English

Health

A suite average hides very different sample sizes, metrics and uncertainty. Small tasks such as RTE and MRPC have high seed variance, while QQP and MNLI dominate training volume but still receive only one task's weight each.

Status evidence

RoBERTa reached 88.5 on the official nine-task leaderboard by 2019-07-25, above the conservative 87.1 non-expert human aggregate; SuperGLUE was introduced for more headroom.

Saturated since

2019-07-25

Refresh cycle

static

Test set public

no

Four constituent tasks originally used private test data and official submissions hide labels, but all training/development data, several test sets, task sources and the diagnostic set are static and widely mirrored.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official GLUE leaderboardofficial