SWE-bench

What it measures

Real GitHub issues from 12 Python repositories, solved by generating patches that must pass repository tests.

Primary metric

resolved instances (%)

Judge model

not judged by a model

Human baseline

—not reportedNo protocol-comparable aggregate human score was published. Every instance has a human-authored pull request, but that does not establish a timed blind-test pass rate.

Languages

English, Python

Health

At 2,294 items, a 50% resolve rate has an approximate 95% binomial interval of ±2.0 points; task and repository clustering makes a simple binomial interval optimistic.

Status evidence

SWE-bench Verified superseded the original test set in 2024 after human review found underspecified tasks and tests that rejected valid fixes.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Evaluation starts from a historical base commit, but issues, merged pull requests, gold patches, and tests are public in repository history.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official SWE-bench leaderboardsofficial