AV-SpeakerBench

What it measures

Speaker-centric audiovisual benchmark for aligning who speaks, what is said, and when in real multi-person videos.

Primary metric

micro-averaged multiple-choice accuracy (%)

Judge model

not judged by a model

Human baseline

93.74Paper-reported human overall accuracy under the benchmark protocol.

Languages

English

Health

Overall accuracy can hide fusion failures because task frequencies differ and some items remain inferable from natural visual speech cues. Report per-task results and visual-only/audio-only ablations alongside the audiovisual score.

Status evidence

A new 2025 benchmark with a 20.7-point gap between the paper's strongest model and human performance, plus large audio-versus-vision ablation gaps across 12 tasks.

Saturated since

not saturated

Refresh cycle

static

Test set public

yes

Expert curation, cross-modal question construction, temporal-sensitivity checks, and unimodal filtering reduce shortcuts at release; the public dataset can still enter later training corpora.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

• Official AV-SpeakerBench projectofficial