XSTest

What it measures

A 450-prompt test of whether models refuse unsafe requests without over-refusing superficially similar safe ones.

Primary metric

unsafe-prompt total refusal rate (%)

Judge model

not judged by a model

Human baseline

—not reportedNo single human score is appropriate. Humans created and labeled the prompts and response taxonomy; the desired direction is compliance on safe prompts and refusal on unsafe prompts.

Languages

English

Health

With 250 safe and 200 unsafe prompts, a few responses move category rates visibly. Timeline scores are manual total refusal on unsafe prompts (full plus partial); the model label records the companion safe-prompt total-refusal rate. Timeline dates are model public-launch dates, not evaluation dates.

Status evidence

XSTest remains a useful two-sided calibration diagnostic because it measures exaggerated refusal alongside harmful compliance, though its small public set should be paired with broader tests.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Keep the official prompt taxonomy intact, publish per-type results, and supplement it with fresh private borderline prompts.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory