StereoSet

What it measures

English context-association tests that separate stereotypical preference from basic language-modeling ability.

Primary metric

ICAT score (0-100; higher balances language modeling and stereotype neutrality)

Judge model

not judged by a model

Human baseline

—not reportedNo protocol-comparable human ICAT baseline is reported. ICAT 100 describes an idealized model with perfect language modeling and stereotype score 50, not measured human performance.

Languages

English

Health

The released Hugging Face validation data contain 2,123 intersentence and 2,106 intrasentence contexts. The often-quoted roughly 17,000 figure counts candidate sentences, not independent scored contexts. ICAT can hide different LMS/SS trade-offs, so publish all three values per track. Timeline dates are model public-launch dates, not evaluation dates.

Status evidence

StereoSet remains a useful historical bias diagnostic, but its static public data, US-centric stereotypes, and sensitivity to scoring choices make it a weak standalone frontier safety benchmark.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Use the released set as a regression diagnostic, disclose the exact split and tokenizer scoring method, and pair it with newer or privately held bias probes.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• StereoSet projectofficial