SWE-Lancer

What it measures

Current 198-task offline patch benchmark derived from paid Upwork work, with historical mixed implementation and manager-task variants.

Primary metric

pass@1 accuracy on the 198-task Diamond offline subset (%)

Judge model

not judged by a model

Human baseline

—not reportedNo blind protocol-comparable aggregate human baseline is reported. Original freelancers completed the IC work and hired managers chose proposals, defining the references rather than a timed test score.

Languages

English, JavaScript, TypeScript

Health

The current offline leaderboard has 198 executable patch tasks. The original paper's 502-task Diamond mixed 237 IC and 265 manager tasks, so its overall accuracy is a different axis.

Status evidence

The maintained leaderboard is frozen at July 2025 and narrows paper-era Diamond from 502 mixed tasks to 198 offline IC tasks, making unlabeled Diamond claims non-comparable.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The benchmark uses a fixed historical codebase and, since July 2025, removes live Internet dependencies from execution; the public Diamond tasks and code are nevertheless exposed.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official SWE-Lancer leaderboardofficial