MBPP

What it measures

Mostly Basic Programming Problems: 974 crowd-sourced Python synthesis tasks intended for entry-level programmers.

Primary metric

pass@1 on the 500-task test split (%)

Judge model

not judged by a model

Human baseline

—not reportedNo controlled aggregate human baseline was published. Tasks were designed to be solvable by entry-level programmers, and a subset was edited and hand-verified, but neither is a measured score.

Languages

Python

Health

At 90% over 500 tasks, the rough 95% binomial interval is about ±2.6 points; protocol and test-quality variation can be larger. Timeline dates are model launch dates; source_url values point to the CodeSOTA MBPP pass@1 registry snapshot.

Status evidence

CodeSOTA's 2026 MBPP pass@1 registry has multiple frontier and open-code models clustered from 89% to 95%, while the short public tasks and three-test protocol remain too weak for current frontier comparison.

Saturated since

2023-11-24

Refresh cycle

static

Test set public

yes

The paper found little overlap with its own pretraining corpus, but all prompts, reference code, and assertions are now public and static.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• CodeSOTA MBPP leaderboardthird-party
• EvalPlus leaderboardthird-party