Tool Decathlon / Toolathlon

What it measures

A long-horizon tool-use benchmark with 108 tasks across 32 software applications, 604 MCP tools, and state-based evaluators.

Primary metric

Pass@1 on Toolathlon-Verified (%)

Judge model

not judged by a model

Human baseline

—not reportedNo single protocol-comparable human baseline is published for the full Toolathlon-Verified set.

Languages

English, Python, Shell, SQL

Health

Toolathlon-Verified starts a new official score series. Do not compare Verified results directly with the archived pre-Verified leaderboard.

Status evidence

Toolathlon-Verified is the current benchmark release as of 2026-06-30. The official site reports 108 tasks, 32 MCP servers with 604 tools, 7 local toolkits, and a current Verified leaderboard.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Toolathlon-Verified reviewed prompts, ground truths, evaluators, initial states, and infrastructure; state-based scripts improve grading reliability but do not hide public tasks.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Toolathlon model leaderboardofficial