Why this benchmark is useful
Editorial brief pendingWe publish the methodology and ledger first; benchmark-specific analysis ships after desk review.
Benchmarks / Multimodal
Massive Multi-discipline Multimodal Understanding
MMMU
College-level questions that require reading images alongside text — charts, diagrams, chemical structures, medical scans — across 30 subjects.
MultimodalSolidActiveMedium contamination riskSince 2023
What this does not measure
- Pure vision quality — many items can be partly solved from the text alone, inflating scores.
- Real document workflows — it is an exam, not 'read this 40-page PDF and extract the table'.
Coverage map
- Task family
- Multimodal
- Format
- ~11,500 multimodal questions (image + text), multiple-choice and open-ended.
- Scoring
- Accuracy (% correct).
- Maintainer
- Yue et al.
How to read the scores
Reading guide pending. Use task format, scoring method, and source dates in the ledger until the desk brief ships.
What it does not cover
- Pure vision quality — many items can be partly solved from the text alone, inflating scores.
- Real document workflows — it is an exam, not 'read this 40-page PDF and extract the table'.
Scores
8 rowsEvidence ledger
88 rows
72.4%best score
2source-checked
1sources
2026-05-28to 2023-11-27
Where the rows land
0255075100
Normalized to this benchmark's axis (0–100). Open the table for raw units.
Newest receipts
- Claude Opus 4.870.6% · MMMU leaderboard
- Gemini 3.5 Flash68.9% · MMMU leaderboard
- GPT 5.571.8% · MMMU leaderboard
- Llama 4 Maverick61.4% · MMMU leaderboard
- Gemini 3.1 Pro72.4% · MMMU leaderboard
Scores use this benchmark's own unit and axis, not a universal quality score.
| # | Model | Release date | Score | Provenance | Trust |
|---|---|---|---|---|---|
| 1 | Gemini 3.1 ProGoogle | 2026-02-19 | 72.4% | MMMU leaderboard2026-02-19 | Needs audit |
| 2 | GPT 5.5OpenAI | 2026-04-23 | 71.8% | MMMU leaderboard2026-04-23 | Needs audit |
| 3 | Claude Opus 4.8Anthropic | 2026-05-28 | 70.6% | MMMU leaderboard2026-05-28 | Needs audit |
| 4 | Gemini 3.5 FlashGoogle | 2026-05-19 | 68.9% | MMMU leaderboard2026-05-19 | Needs audit |
| 5 | Claude Sonnet 4.6Anthropic | 2026-02-17 | 66.2% | MMMU leaderboard2026-02-17 | Needs audit |
| 6 | Llama 4 MaverickMeta | 2026-04-05 | 61.4% | MMMU leaderboard2026-04-05 | Needs audit |
| 7 | Gemini 1.0 UltraGoogle | 2024-02-08 | 59.4% | Gemini technical report2023-12-06 | Source-checked |
| 8 | GPT-4V (2023)OpenAI | 2023-09-25 | 56.8% | MMMU paper2023-11-27 | Source-checked |
What it covers
Data-quality note pending. Every ledger row still carries source URL, source date, and ingest timestamp.
Where it breaks down
Human experts ~88%.
Receipts
Sources and further reading
- Paper (Yue et al.)· checked 2026-05-30
- Leaderboard
The Pack · Editorial newsletter
New cards in your inbox. Free.
One short email when a card ships or changes status. No tracking, no third-party analytics. Unsubscribe in one click.