SWE-bench Pro

What it measures

Long-horizon repository tasks across public, held-out, and commercial codebases, with a continuing public model leaderboard.

Primary metric

resolved instances on the public set (%)

Judge model

not judged by a model

Human baseline

—not reportedNo protocol-comparable aggregate human score is published. The authors describe tasks as requiring hours to days for professionals, and tasks have human reference solutions.

Languages

English, Go, JavaScript, Python, TypeScript

Health

At 731 items and 50% resolution, the nominal 95% binomial interval is about ±3.6 points. The 2026 audit's estimated ~30% broken-task rate dominates that sampling uncertainty. Dates on the 17 newest timeline points denote the 2026-08-03 LLM Stats snapshot, not individual evaluation or model-launch dates.

Status evidence

The benchmark still has a maintained public leaderboard and substantial score spread: the 2026-08-03 LLM Stats snapshot ranges from 0.584 to 0.800 across these 17 leading models, despite documented task-quality concerns.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Public and held-out OSS repositories use strong-copyleft licenses as a legal deterrent to training inclusion; the commercial split uses private proprietary code. Neither is a technical guarantee.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official public leaderboardofficial
• Official commercial leaderboardofficial
• LLM Stats SWE-bench Pro leaderboardthird-party