Skip to content
LLM Atlas

Benchmarks

SWE-bench Verified

saturatedCodingcontamination risk: highlaunched 2024-08-13

Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.

What it measures

A 500-task, engineer-reviewed subset of SWE-bench that became the standard repository-level coding-agent test.

Primary metric
resolved instances (%)
Judge model
not judged by a model
Human baseline
not reportedNo protocol-comparable aggregate human pass rate was published. Annotators estimated fix time, and each task has a human-authored pull request, but neither is a blind human baseline.
Languages
English, Python

Health

At 500 items, a 50% resolve rate has an approximate 95% binomial interval of ±4.4 points; repository clustering, flaky environments, and test defects add non-sampling uncertainty.

Status evidence
Classified saturated on 2026-02-17: under the standardized mini-SWE-agent 2.0.0 protocol, the leading results clustered at 76.8%, 75.8%, 75.8%, and 75.6%, limiting discrimination among frontier systems; separate contamination and task-quality concerns remain documented below.
Saturated since
2026-02-17
Refresh cycle
static
Test set public
yes

Three professional developers screened each candidate for issue and test quality, but the issues, gold patches, and repository histories remain public and static.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

Independently measured

ModelScoreHarnessMeasuredProvenanceHealth
Claude Opus 4.7 (max)Anthropic83.5%not reported2026-04-16ind.saturatedcontaminated
gpt-5.5-pre-release (xhigh)OpenAI80.6%not reported2026-04-23ind.saturatedcontaminated
Gemini 3.5 Flash (high)Google DeepMind79.3%not reported2026-05-19ind.saturatedcontaminated
Claude Opus 4.6Anthropic78.7%not reported2026-02-05ind.saturatedcontaminateddisputed
GLM-5.2 (max)Z.ai (Zhipu AI)78.7%not reported2026-06-16ind.saturatedcontaminated
Qwen3.7-MaxAlibaba77.3%not reported2026-05-19ind.saturatedcontaminated
GPT-5.4 (high)OpenAI76.9%not reported2026-03-05ind.saturatedcontaminated
Qwen3.6-Max (preview)Alibaba76.7%not reported2026-04-20ind.saturatedcontaminated
Kimi K2.6Moonshot76.7%not reported2026-04-20ind.saturatedcontaminated
Claude Opus 4.5Anthropic76.7%not reported2025-11-24ind.saturatedcontaminated
Gemini 3.1 Pro Preview Custom ToolsGoogle DeepMind75.6%not reported2026-02-19ind.saturatedcontaminated
Claude Opus 4.6Anthropic75.6%not reported2026-02-05ind.saturatedcontaminateddisputed
Gemini 3 Flash PreviewGoogle DeepMind75.4%not reported2025-12-17ind.saturatedcontaminated
Claude Sonnet 4.6Anthropic75.2%not reported2026-02-17ind.saturatedcontaminated
GPT-5.3-Codex (high)OpenAI74.8%not reported2026-02-05ind.saturatedcontaminated
Glm 5.1Z.ai (Zhipu AI)74.2%not reported2026-04-07ind.saturatedcontaminated
Kimi K2.5Moonshot73.8%not reported2026-01-27ind.saturatedcontaminated
GPT-5.2 (high)OpenAI73.8%not reported2025-12-11ind.saturatedcontaminated
GPT-5 (high)OpenAI73.6%not reported2025-08-07ind.saturatedcontaminated
Claude Opus 4.1Anthropic73.3%not reported2025-08-05ind.saturatedcontaminated
Gemini 3 ProGoogle DeepMind72.9%not reported2025-11-18ind.saturatedcontaminated
GLM 5Z.ai (Zhipu AI)72.1%not reported2026-02-11ind.saturatedcontaminated
GPT-5 (medium)OpenAI71.5%not reported2025-08-07ind.saturatedcontaminated
Claude Sonnet 4.5Anthropic71.3%not reported2025-09-29ind.saturatedcontaminated
Claude Opus 4Anthropic70.7%not reported2025-05-22ind.saturatedcontaminated
GPT-5.1 (high)OpenAI68.0%not reported2025-11-13ind.saturatedcontaminateddisputed
GPT-5.1 (high)OpenAI65.9%not reported2025-11-13ind.saturatedcontaminateddisputed
GPT-5 Mini (medium)OpenAI64.7%not reported2025-08-07ind.saturatedcontaminated
o3 (medium)OpenAI62.3%not reported2025-04-16ind.saturatedcontaminated
Claude 3.7Anthropic61.0%not reported2025-02-24ind.saturatedcontaminated
Qwen3.6-PlusAlibaba57.9%not reported2026-03-31ind.saturatedcontaminated
Gemini 2.5 ProGoogle57.6%not reported2025-06-17ind.saturatedcontaminated
GPT-4.1OpenAI48.5%not reported2025-04-14ind.saturatedcontaminated
GPT-4oOpenAI31.0%not reported2024-11-20ind.saturatedcontaminated

Trajectory

Best score measured on this site

0255075100GPT-4o: 31.0%Claude 3.7: 61.0%GPT-4.1: 48.5%o3: 62.3%Claude Opus 4: 70.7%Gemini 2.5 Pro: 57.6%Claude Opus 4.1: 73.3%GPT-5: 73.6%GPT-5: 71.5%GPT-5 Mini: 64.7%Claude Sonnet 4.5: 71.3%GPT-5.1: 68.0%GPT-5.1: 65.9%Gemini 3 Pro: 72.9%Claude Opus 4.5: 76.7%GPT-5.2: 73.8%Gemini 3 Flash Preview: 75.4%Kimi K2.5: 73.8%Claude Opus 4.6: 78.7%Claude Opus 4.6: 75.6%GPT-5.3-Codex: 74.8%GLM 5: 72.1%Claude Sonnet 4.6: 75.2%Gemini 3.1 Pro Preview Custom Tools: 75.6%GPT-5.4: 76.9%Qwen3.6-Plus: 57.9%Glm 5.1: 74.2%Claude Opus 4.7: 83.5%Qwen3.6-Max (preview): 76.7%Kimi K2.6: 76.7%gpt-5.5-pre-release: 80.6%Gemini 3.5 Flash: 79.3%Qwen3.7-Max: 77.3%GLM-5.2: 78.7%GPT-4o: 31.0%, best at the timeClaude 3.7: 61.0%, best at the timeo3: 62.3%, best at the timeClaude Opus 4: 70.7%, best at the timeClaude Opus 4.1: 73.3%, best at the timeGPT-5: 73.6%, best at the timeClaude Opus 4.5: 76.7%, best at the timeClaude Opus 4.6: 78.7%, best at the timeClaude Opus 4.7: 83.5%, best at the time20242026
Each dot is a score measured on this site; the line steps up at each new best and stays flat until the next one. A line that has stopped climbing is what saturation looks like.

As benchwiki records it

GPT-4o (Agentless) — 33.2 on 2024-08-13 (vendor-reported)Claude 3.5 Sonnet (upgraded; Anthropic tools scaffold) — 49 on 2024-10-22 (vendor-reported)Claude 4.5 Opus (high reasoning; mini-SWE-agent 2.0.0) — 76.8 on 2026-02-17 (independent)Gemini 3 Flash (high reasoning; mini-SWE-agent 2.0.0) — 75.8 on 2026-02-17 (independent)MiniMax M2.5 (high reasoning; mini-SWE-agent 2.0.0) — 75.8 on 2026-02-17 (independent)Claude Opus 4.6 (mini-SWE-agent 2.0.0) — 75.6 on 2026-02-17 (independent)GPT-5-2 Codex (mini-SWE-agent 2.0.0) — 72.8 on 2026-02-19 (independent)GLM-5 (high reasoning; mini-SWE-agent 2.0.0) — 72.8 on 2026-02-17 (independent)GPT-5-2 (high reasoning; mini-SWE-agent 2.0.0) — 72.8 on 2026-02-17 (independent)GPT 5.2 Codex (mini-SWE-agent 2.0.0) — 72.8 on 2026-02-19 (independent)Claude 4.5 Sonnet (high reasoning; mini-SWE-agent 2.0.0) — 71.4 on 2026-02-17 (independent)Kimi K2.5 (high reasoning; mini-SWE-agent 2.0.0) — 70.8 on 2026-02-17 (independent)DeepSeek V3.2 (high reasoning; mini-SWE-agent 2.0.0) — 70 on 2026-02-17 (independent)Gemini 3 Pro (mini-SWE-agent 2.0.0) — 69.6 on 2026-02-26 (independent)Claude 4.5 Haiku (high reasoning; mini-SWE-agent 2.0.0) — 66.6 on 2026-02-17 (independent)GPT-5 Mini (mini-SWE-agent 2.0.0) — 56.2 on 2026-02-17 (independent)20242026770
independent · vendor-reported. Points are not joined: the scores come from different models, harnesses and protocol versions, and a line between them would claim a continuity the data does not have.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

Source

This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2026-08-03; fetched 2026-10-19 by this site.