StructFlowBench

What it measures

155 multi-turn dialogues scored on fine-grained constraints plus six inter-turn dialogue relationships (recall, refinement, expansion, follow-up, summary).

Primary metric

WCSR — Weighted Constraint Satisfaction Rate (%)

Judge model

GPT-4o (the paper's automated evaluator — see scoring methodology for the reproducibility caveat)

Human baseline

—not reportedNo protocol-comparable human baseline published. The 'Golden Context' setup drops each model into the same curated dialogue history rather than one it generated, but the paper reports no human agreement with the GPT-4o judge on model outputs.

Languages

English

Health

155 dialogues and a per-constraint score grid under a single LLM judge; per-turn and per-relationship cells are small. Treat WCSR gaps under a few points as noise until a leaderboard reports confidence intervals or a second judge.

Status evidence

Moved to the site's nearing-saturation bucket: the best reported WCSR is already about 0.96 on a 155-dialogue public static set, while the remaining signal is concentrated in harder relationship slices such as refinement rather than the headline score.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The dataset was released on 2025-02-20 and postdates most 2024 training cutoffs, so direct memorization is currently less of a risk. The structural relationships are a behavioral/semantic target rather than a fixed factual test set, so memorizing one dialogue does little to help on unseen dialogue flows.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official StructFlowBench leaderboard (in-repo image)official