MathVista

What it measures

Visual mathematical reasoning across diagrams, plots, tables, scientific figures, puzzles, and natural images.

Primary metric

accuracy (%)

Judge model

GPT-4 answer extractor in the original protocol

Human baseline

60.3Amazon Mechanical Turk average on testmini; annotators had a high-school diploma or higher. This is not an expert-mathematician ceiling.

Languages

English

Health

Leaderboard comparisons must use the same split, prompt, answer extractor, and tool setting. A 1,000-item testmini accuracy near 50% has a rough 95% binomial margin of about +/-3 points before accounting for heterogeneous tasks.

Status evidence

The official leaderboard reported multiple models above the 60.3% crowd-worker baseline by 2024; the public 1,000-item testmini remains useful for diagnosis but is no longer a clean frontier-only signal.

Saturated since

not saturated

Refresh cycle

static

Test set public

no

The 5,141 test labels were withheld at release, but all testmini labels and source images/questions are public; many items are inherited from 28 existing datasets.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official MathVista leaderboardofficial