Soohak

What it measures

Fresh research-level mathematics authored by mathematicians, plus unanswerable prompts that reward justified refusal.

Primary metric

Challenge Avg@3 (%)

Judge model

GPT-5-Mini

Human baseline

—not reportedFive invited human teams collectively solved 50.6% of a 79-prompt sample, but this coverage result is not protocol-comparable to a single model's Avg@3.

Languages

English, Korean

Health

Challenge has 340 items and Refusal 99. Three samples reduce decoding noise but do not make Avg@3 a single-pass accuracy; judge error and subject mix remain material.

Status evidence

The official July 2026 tracker reports 64.4% Avg@3 for the Challenge leader with 46 of 340 problems still unsolved by any tracked model; Refusal remains a separate hard target.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

Problems were newly authored without AI assistance under confidentiality controls; the full collection is embargoed until late 2026 and models are evaluated by request.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official Soohak leaderboardofficial