CodeContests

What it measures

DeepMind's competitive-programming corpus with temporally split problems, human submissions, and generated correctness tests.

Primary metric

10@100k solve rate on the 165-problem test split (%)

Judge model

not judged by a model

Human baseline

—not reportedNo human 10@100k solve-rate baseline was published. AlphaCode's human comparison was average simulated rank in the top 54.3% across ten Codeforces contests, an incompatible percentile that must not be plotted on the solve-rate axis.

Languages

C++, Java, Python

Health

The 165-problem test set gives wide intervals for solve rate; the original 10@100k pipeline also introduces sampling, filtering, clustering, and execution variance.

Status evidence

The repository was archived on 2024-12-06, the temporal holdout has been public since 2022, and later AlphaCode systems used updated non-public data; it no longer provides a clean maintained frontier comparison.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

The original temporal split placed training data before validation/test problems, but the complete released split has been public since 2022 and the clean window has expired.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory