MM-SafetyBench

What it measures

A 5,040-pair benchmark testing whether query-relevant images can bypass multimodal model safeguards.

Primary metric

attack success rate (%) - lower is safer

Judge model

GPT-4 in the paper protocol

Human baseline

—not reportedNo protocol-comparable human attack-success baseline is reported; humans are not safety-aligned multimodal target models.

Languages

English

Health

ASR depends on the scenario, image variant, target model, model safety policy, prompt formatting, and GPT-4 judge implementation. Timeline points use the comparable tiny-set SD+TYPO condition; lower is safer. Timeline dates are target-model public-launch dates, not evaluation dates.

Status evidence

MM-SafetyBench established the image-based jailbreak threat, but its static public scenarios and early-model protocol have been overtaken by broader multimodal safety suites and later variants.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Version question and image files, keep image conditions separate, use response-only judging where possible, and add private image attacks generated after the target model's training cutoff.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory