Terminal-Bench

What it measures

Eighty-nine realistic terminal tasks graded in containers; version 2.1 repairs 2.0's dependency, resource, and specification defects.

Primary metric

task accuracy on Terminal-Bench 2.1 (%)

Judge model

not judged by a model

Human baseline

—not reportedNo aggregate blind human baseline is published. Every 2.0 task has a human-written solution and expert-time metadata, establishing solvability but not a comparable score.

Languages

C, C++, English, JavaScript, Python, Rust, Shell

Health

With only 89 tasks, one task changes accuracy by about 1.12 points and a 50% pass rate has a nominal 95% interval near ±10.4 points. Agent/model pair, trial count, resources, and task version are essential.

Status evidence

Terminal-Bench 2.1 remains actively maintained and submission-driven; its verified 17-entry leaderboard spans 58.7% to 83.8% across multiple agent-model pairs, while Terminal-Bench 3.0 is still in development.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Task repositories include a canary asking training pipelines to exclude benchmark data; 2.1 adds continuous dependency and oracle validation, but neither prevents public-task memorization.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Trajectory

Best score measured on this site

As benchwiki records it

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official Terminal-Bench 2.1 leaderboardofficial