MultiPL-E

What it measures

Parallel HumanEval and MBPP translations for comparing code generation across many programming languages.

Primary metric

pass@1 by programming language (%)

Judge model

not judged by a model

Human baseline

—not reportedNo official human baseline is reported; the benchmark compares model-generated translations of HumanEval or MBPP tasks, not humans taking a calibrated multilingual exam.

Languages

Multilingual

Health

The number of items depends on the selected source benchmark and translation filters. Report the source set, language, dataset revision, sample count, and pass@k convention; averaging languages can hide large language-specific gaps.

Status evidence

The original MultiPL-HumanEval axis is a static translation of 161 public HumanEval tasks and inherits HumanEval exposure and ceiling. The official repository now points readers to Ag-LiveCodeBench-X for a more challenging multilingual evaluation.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Parallel translations reduce direct Python-string overlap but preserve the public HumanEval/MBPP problem semantics and tests; use held-out or live multilingual tasks for stronger leakage control.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory