HumanEval

What it measures

OpenAI's 164 hand-written Python function-synthesis tasks, graded by executing generated completions against unit tests.

Primary metric

pass@1 (%)

Judge model

not judged by a model

Human baseline

—not reportedNo controlled aggregate human baseline was published; the tasks were hand-written and reference solutions were authored by people, but that is not a timed pass@1 measurement.

Languages

Python

Health

With 164 binary tasks, a score near 90% has a rough 95% binomial interval of about ±4.6 points before accounting for sampling and prompt variance.

Status evidence

Modern coding models routinely cluster near the ceiling on the small public set, while EvalPlus showed the original sparse tests accept materially incorrect programs.

Saturated since

2023-05-02

Refresh cycle

static

Test set public

yes

The authors hand-wrote problems to avoid direct overlap with their training data, but prompts, tests, and canonical solutions have been public since 2021.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• EvalPlus leaderboardthird-party