HumanEval+

What it measures

HumanEval's 164 Python tasks re-evaluated with roughly 80 times more tests to expose plausible but incorrect programs.

Primary metric

HumanEval+ pass@1 (%)

Judge model

not judged by a model

Human baseline

—not reportedNo controlled aggregate human pass@1 baseline has been published for HumanEval+; human-written canonical solutions are test oracles, not a timed baseline.

Languages

Python

Health

Only 164 task-level outcomes are averaged; near 90%, the rough 95% binomial interval is about ±4.6 points, separate from evaluator-version and generation variance.

Status evidence

The official EvalPlus leaderboard reports o1-preview and o1-mini at 89.0% HumanEval+ pass@1; with only 164 fully public prompts, remaining score gaps are too small and exposure-prone for a reliable frontier ranking.

Saturated since

2024-09-12

Refresh cycle

static

Test set public

yes

Extra tests make shallow overfitting harder, but the prompts, canonical solutions, and generated tests are public and versioned rather than continuously refreshed.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official EvalPlus leaderboardofficial