FireBench

What it measures

Evaluates instruction following for enterprise/API pipelines — exact output format, strict step ordering, ranking, and calibrated refusal across six categories.

Primary metric

composite instruction-following score

Judge model

not judged by a model

Human baseline

—not reportedNo human baseline published. The categories map to concrete enterprise requirements, but the paper reports only model performance.

Languages

English

Health

2,470 samples across six categories → roughly ±2pt per category at ~50% accuracy (95% CI) if categories are evenly sized; the two content-requirement categories graded by judges carry extra variance from judge disagreement.

Status evidence

Newly released (arXiv paper submitted 5 March 2026); no refresh cycle or successor yet, and scores still separate frontier models. Worth tracking but with no independent replications published so far.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The benchmark is open-sourced at fire-bench.com and in the FireBenchIF/FireBench GitHub repo (CC-BY-4.0), and released a canary string discouraging inclusion in training corpora. However, the distributed data is a partial subset that is regenerated procedurally by the harness, and the paper's 1,300/2,470 remarked "test instances" are not pre-materialized. The categories target behavior (exact formats, ordering, refusal), which is harder to memorize than factual answers.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

No trajectory is published for this benchmark upstream, so none is shown. An empty chart would imply the scores exist and are flat.