Skip to content
LLM Atlas

Benchmarks

QMSum

nearing saturationLong contextcontamination risk: highlaunched 2021-04-13

Benchmark metadata from benchwiki — this page mirrors their record so scores can be joined to it. The original is canonical.

What it measures

Query-focused meeting summarization over long transcripts from academic, product, and committee meetings.

Primary metric
ROUGE over query-focused summaries
Judge model
not judged by a model
Human baseline
not reportedThe dataset is human-annotated, but no single protocol-comparable human generation score is recorded here.
Languages
English

Health

ROUGE is sensitive to wording and reference-summary coverage; small score gaps should not be over-read without human inspection. The visual graph on this page deliberately plots only the official QMSum paper / repository Table 3 ROUGE-1 (R-1) scores. It does not plot R-2 or R-L.

Status evidence
QMSum is still useful as a query-focused summarization component, but the public static 2021 standalone task is now mainly a subtask inside SCROLLS, ZeroSCROLLS, and LongBench-style suites.
Saturated since
not saturated
Refresh cycle
static
Test set public
yes

The dataset is public and static, and its meeting transcripts are distributed in the official repository.

Models scored on this benchmark

The reverse lookup, which is this site’s addition: every model here with a score on this benchmark. Independent measurements and vendor-reported numbers are kept apart rather than interleaved, because they are different claims.

No model on this site has a score on this benchmark yet. That usually means the models that were measured on it are not in the registry, rather than that nobody has run it.

Trajectory

Random (paper Table 3 R-1) — 12.03 on 2021-04-13 (independent)Ext. Oracle (paper Table 3 R-1) — 42.84 on 2021-04-13 (independent)TextRank (paper Table 3 R-1) — 16.27 on 2021-04-13 (independent)PGNet (paper Table 3 R-1) — 28.74 on 2021-04-13 (independent)BART (paper Table 3 R-1) — 29.2 on 2021-04-13 (independent)PGNet* (paper Table 3 R-1) — 31.37 on 2021-04-13 (independent)BART* (paper Table 3 R-1) — 31.74 on 2021-04-13 (independent)HMNet* (paper Table 3 R-1) — 32.29 on 2021-04-13 (independent)PGNet (gold spans) (paper Table 3 R-1) — 31.52 on 2021-04-13 (independent)BART (gold spans) (paper Table 3 R-1) — 32.18 on 2021-04-13 (independent)HMNet (gold spans) (paper Table 3 R-1) — 36.06 on 2021-04-13 (independent)20212021430
independent · vendor-reported. Points are not joined: the scores come from different models, harnesses and protocol versions, and a line between them would claim a continuity the data does not have.

Where results are tracked

Leaderboards benchwiki lists for this benchmark. These are links out, not sources this site ingests — no score below comes from any of them.

Source

This record is maintained by benchwiki and mirrored here. Read the original. Upstream last updated 2021-04-13; fetched 2026-10-19 by this site.