AgentIF
Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.
What it measures
707 instructions from 50 real-world agents, averaging 1,723 words and 11.9 constraints — whether instruction following survives deployment-scale prompts.
- Primary metric
- CSR (Constraint Success Rate, %)
- Judge model
- Evaluator LLM (paper uses GPT-4o / gpt-4o-2024-11-20 for the LLM-based and hybrid constraint checks; some constraints use pure code verification, so not every item passes through the judge)
- Human baseline
- not reportedNo protocol-comparable aggregate human baseline is reported. Annotations were by humans, so ISR at full annotation agreement is implicit; the paper does not publish a 'human would score X' figure.
- Languages
- Chinese, English
Health
707 instructions → an ISR near 27% has a rough 95% binomial interval of about ±3 points; constraints are nested inside instructions, so treating items as independent understates the variance.
- Status evidence
- Moved to the site's nearing-saturation bucket: the benchmark is still useful for exposing agentic constraint failures, but the public static set is small, top CSR is already near 60%, and the main value is shifting from broad frontier ranking to targeted failure analysis.
- Saturated since
- not saturated
- Refresh cycle
- static
- Test set public
- yes
The prompts are drawn from real deployed agents and open-source workflows and are public on HuggingFace since release; nothing is held out or refreshed. Note the instructions themselves are real system prompts, so a model may have already seen structurally similar living workflows in pretraining — but the specific query/constraint pairings are bespoke.
Models scored on this benchmark
The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.
No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.
Trajectory
Where results are tracked
Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.
Source
This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2026-08-12; fetched 2026-10-19 by this site.