MT-Bench

What it measures

80 curated multi-turn questions across 8 skills, each a two-turn conversation scored by a strong LLM judge on a 1–10 scale.

Primary metric

average judge score (1–10)

Judge model

GPT-4 (original paper; later reported with GPT-4-family judges, so cross-time scores are comparable only when the judge is the same)

Human baseline

—not reportedNo canonical human score on the 1–10 judge scale has been published. The paper instead validated the judge against human preference annotations: GPT-4-judge agreement with human preference exceeded 80%, roughly matching human–human agreement on the same annotations. That is an agreement check, not a human competence score.

Languages

English

Health

80 two-turn questions, with 4 or fewer per category for three of the eight categories, is small enough that reported top-model gaps of a few tenths of a point on a 1–10 scale are close to noise; treat sub-~0.3 point differences (and any single-category comparison) with caution.

Status evidence

The 80 two-turn questions are fixed, public, and heavily exercised across research and leaderboards, so the test set is both widely memorized and increasingly saturated (top models score 8.5–9.0 on a 1–10 scale where 9 is a realistic ceiling). It remains a standard, still-discriminative LLM-judge eval, but the fixed set is the constraint: refreshed or held-out variants are the future direction.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The 80 questions are fixed, shipped inside the FastChat repo, and have been used heavily across research, leaderboards, and (per several labs) included in instruction-tuning and alignment data mixtures. There is no refreshed or held-out MT-Bench; the classification question is therefore mostly fairness across contemporary models on the same fixed set.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory