MBPP+

What it measures

A cleaned 378-task MBPP evaluation with roughly 35 times more tests for stricter Python functional correctness.

Primary metric

MBPP+ pass@1 (%)

Judge model

not judged by a model

Human baseline

—not reportedNo controlled human pass@1 baseline is published for the cleaned MBPP+ set; reference implementations serve as test oracles, not a timed human evaluation.

Languages

Python

Health

At 90% over 378 tasks, the rough 95% binomial interval is about ±3.0 points. The pre-v0.3.0 removal of 21 broken tasks changed expected scores by about four points. Timeline dates are official model launch dates; scores come from BenchmarkList's 2026-07-13 snapshot and are not claimed as launch-day evaluations.

Status evidence

The 2026-07-13 BenchmarkList snapshot tops out at 80.2% pass@1, while the small public task set, provider-sourced reporting, and dataset-version sensitivity limit clean frontier discrimination.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Expanded tests catch more overfitting, but both the inherited MBPP prompts and MBPP+ tests are public and versioned.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official EvalPlus leaderboardofficial
• BenchmarkList MBPP+ leaderboard snapshotthird-party