JailbreakBench

What it measures

An open framework, behavior set, artifact repository, and leaderboard for comparing jailbreak attacks and defenses.

Primary metric

attack success rate on 100 misuse behaviors (%)

Judge model

Llama-3-70B jailbreak judge in the current official package

Human baseline

—not reportedNo human attack-success baseline is defined. A separate 300-example judge-comparison set contains three human labels per response for judge validation.

Languages

English

Health

ASR is conditional on target model version, system prompt, chat template, attack budget, defense, generation settings, and judge. Timeline points use the paper's undefended PAIR attack only so one attack protocol is compared across targets; lower is safer. Timeline dates are target-model public-launch dates, not evaluation dates.

Status evidence

JailbreakBench remains a useful standardized red-teaming framework because it fixes behaviors, threat models, templates, judges, and artifact formats, while accepting evolving attacks and defenses.

Saturated since

not saturated

Refresh cycle

continuous

Test set public

yes

Keep target behaviors fixed for comparability, version jailbreak artifacts and judges, enforce equal target-call budgets, and maintain private behaviors for deployment decisions.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official JailbreakBench leaderboardofficial