CodeXGLUE

What it measures

A broad code-intelligence suite spanning understanding, retrieval, completion, translation, repair, generation, and summarization.

Primary metric

overall smoothed BLEU on six-language CodeSearchNet code summarization

Judge model

not judged by a model

Human baseline

—not reportedNo single human baseline exists across the ten heterogeneous tasks; several component datasets use human-authored code or labels rather than a common human score.

Languages

C, C++, C#, Chinese, English, Go, Japanese, Java, JavaScript, PHP, Python, Ruby

Health

There is no statistically coherent suite-wide score. Compare systems only within the same dataset, split, preprocessing, and metric.

Status evidence

The official challenge and repository remain available, but the heterogeneous task-specific leaderboard is no longer a current frontier-code comparison and has no maintained common aggregate.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

Official test labels were withheld for challenge submission, but most source corpora and all training/dev material are public; component datasets require separate leakage analysis.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official CodeXGLUE leaderboardofficial