Mercury

What it measures

Python code-synthesis benchmark that rewards both functional correctness and runtime efficiency against real solution distributions.

Primary metric

Beyond with 5 sampled solutions per task (%)

Judge model

not judged by a model

Human baseline

—not reportedNo official human Beyond@1 baseline is reported. The comparison reference is a distribution of accepted human-written LeetCode solutions, not people answering the benchmark under controlled conditions.

Languages

Python

Health

Beyond depends on runtime percentiles and the execution environment. Compare models only under the same hardware, sandbox, test generators, timeout policy, sample count, and Mercury revision.

Status evidence

Mercury is a static 256-task public LeetCode-derived evaluation with no official maintained leaderboard or refreshed test set. Beyond remains useful for efficiency research, but the 2024 launch snapshot is no longer a frontier comparison.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The authors deduplicate collected solutions and use generated tests, but tasks derive from public LeetCode problems and the released evaluation data are static.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory