ComplexBench

What it measures

Bilingual benchmark of how models follow instructions when several constraints must hold at once — where stacking constraints is not linear.

Primary metric

overall constraint-satisfaction (%)

Judge model

GPT-4, augmented with rule-based checks where a constraint is verifiable programmatically (e.g. length), per the paper's evaluation protocol

Human baseline

—not reportedNo protocol-comparable human constraint-satisfaction baseline was published. The dataset is manually curated and validated by humans, but the authors do not report a human pass-rate that maps one-to-one onto the automatic judge score.

Languages

Chinese, English

Health

The overall percentage is a composite built over per-instance dependency trees, not a clean binomial hit-rate, so simple confidence intervals mislead. Because many scoring questions gate on prerequisites, one early failure can cascade and suppress several downstream points at once.

Status evidence

Moved to the site's nearing-saturation bucket: ComplexBench remains useful for multi-constraint composition analysis, but it is a public static set and should be treated more as a diagnostic successor to simpler instruction-following checks than as a durable frontier-ranking benchmark.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Items are largely self-contained synthetic compositions written for the benchmark rather than scraped from the internet, so the highest-risk web-memorization leak path is limited. The dataset and gold scoring questions are still public, so fine-tuning on files would circumvent it.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.