Benchmarks that still separate.

AA output speedOutput tokens per second

Use this when · When you care how fast tokens stream after the first chunk — IDE feel, chat responsiveness, batch throughput — not whether the answer is right.

Won't tell you · IDE or chat-app latency — only first-party API routes AA tracks as the default provider.

203 score rows · 1 source · refreshed 2026-11-02

Artificial Analysis Long Context ReasoningAA-LCR

Use this when · Long-context research breaks when models lose thread across distant evidence. AA-LCR is one of the few public suites that grades synthesis across long inputs instead of retrieval trivia alone.

Won't tell you · Retrieval from external databases — the context is in-prompt only.

184 score rows · 1 source · refreshed 2026-09-28

Omniscience AccuracyAA-Omniscience

Use this when · For source-heavy students the failure mode that matters is confident wrong answers. AA-Omniscience is the public suite that jointly scores recall and hallucination across domains that actually show up in papers.

Won't tell you · Open-web research skill — questions are closed-book. A model that should search and cite is not being tested here.

12 score rows · 1 source · refreshed 2026-09-01

CritPtCritical Physics Tasks

Use this when · When GPQA and HLE no longer tell you whether a model can do research-scale physics: original, unpublished problems with guess-resistant answers, still far from ceiling (frontier around 30%).

Won't tell you · Textbook or contest physics — problems are original research-scale tasks, not recycled qualifying-exam items.

12 score rows · 1 source · refreshed 2026-09-01

Use this when · When you need long-horizon, multi-language software work with behavioral verifiers — harder contamination posture than classic SWE-bench Verified.

Won't tell you · Python-only issue fixing like SWE-bench Verified — DeepSWE spans Go, Rust, TypeScript, and more.

8 score rows · 1 source · refreshed 2026-08-04

Use this when · When MATH and AIME no longer separate the frontier, this is the live math discriminator: original expert problems, Python-enabled, still far from ceiling.

Won't tell you · Tier 4 research-level problems — that 43-problem expansion is a separate, harder set. Do not mix Tiers 1-3 headlines with Tier 4.

15 score rows · 1 source · refreshed 2026-08-15

GPQA DiamondGraduate-Level Google-Proof Q&A

Use this when · Use GPQA Diamond when you need a discriminating frontier-science check after easier suites saturate. Treat it as one reasoning receipt among others — useful for model comparison, not as a pick list for student writing tools.

Won't tell you · General usefulness — it is a narrow slice of frontier science, not everyday tasks.

184 score rows · 2 sources · refreshed 2026-09-28

Humanity's Last ExamHLE

Use this when · When you need a hard, cross-domain expert exam — math through professional law and medicine — as a closed-book stress test, not a research-productivity proxy.

Won't tell you · Real research productivity — exam-style questions do not capture the messy search-and-synthesize work a researcher actually does.

186 score rows · 1 source · refreshed 2026-09-28

MMMU-ProMMMU Pro

Use this when · Original MMMU no longer separates the frontier. MMMU-Pro is the live multimodal exam: shortcuts removed, ten options, still a spread below the old 88% ceiling.

Won't tell you · Original MMMU headlines — those still include text-solvable items. A 88%+ MMMU row is not an MMMU-Pro row.

8 score rows · 1 source · refreshed 2026-09-01

SciCode

Use this when · When you care about scientific programming with domain libraries and multi-step numerical code — not general SWE issue fixing.

Won't tell you · Wet-lab validation — generated code is graded against reference tests only.

97 score rows · 1 source · refreshed 2026-09-28

Terminal-BenchTerminal-Bench 2.1

Use this when · When you need realistic containerized terminal work — debug, files, commands, artifacts — as an agentic ops signal. This page tracks AA's published 2.1 / Hard rows, not the v4.0 slice inside Index v4.3.

Won't tell you · The model alone — harness, tools, retries, time limits, and container setup can move scores dramatically.

167 score rows · 2 sources · refreshed 2026-09-28

Use this when · For source-heavy students and researchers this is the live science-agent discriminator: real lab-style workflows, still far from ceiling (best published 0.1 row is 30%).

Won't tell you · The model alone — every published 0.1 row names an agent harness (Claude Code, Codex, Grok Build). Harness choice moves the score.

9 score rows · 1 source · refreshed 2026-09-01