Every benchmark, honestly read.

AA output speedOutput tokens per second

Use this when · When you care how fast tokens stream after the first chunk — IDE feel, chat responsiveness, batch throughput — not whether the answer is right.

Won't tell you · IDE or chat-app latency — only first-party API routes AA tracks as the default provider.

187 score rows · 1 source · refreshed 2026-09-05

AA time to first tokenTTFT

Use this when · Time-to-first-token is the pause before anything appears — useful when snappy starts matter more than sustained generation speed.

Won't tell you · End-to-end task time for agent loops — TTFT is one hop, not a full coding session.

186 score rows · 1 source · refreshed 2026-09-05

Artificial Analysis Long Context ReasoningAA-LCR

Use this when · Long-context research breaks when models lose thread across distant evidence. AA-LCR is one of the few public suites that grades synthesis across long inputs instead of retrieval trivia alone.

Won't tell you · Retrieval from external databases — the context is in-prompt only.

179 score rows · 1 source · refreshed 2026-09-01

Omniscience AccuracyAA-Omniscience

Use this when · For source-heavy students the failure mode that matters is confident wrong answers. AA-Omniscience is the public suite that jointly scores recall and hallucination across domains that actually show up in papers.

Won't tell you · Open-web research skill — questions are closed-book. A model that should search and cite is not being tested here.

12 score rows · 1 source · refreshed 2026-09-01

Use this when · ARC Prize is the only benchmark series that repeatedly surfaces new inflection points — from reasoning systems (ARC-AGI-2) to agentic intelligence (ARC-AGI-3).

Won't tell you · Knowledge or language — tasks are deliberately knowledge-free, so scores say nothing about factual accuracy.

16 score rows · 1 source · refreshed 2026-06-22

Artificial Analysis Intelligence IndexAA Intelligence Index

Use this when · Single-number model rankings hide tradeoffs. The AA Intelligence Index is a weighted composite of public sub-evals — useful as a dated frontier snapshot when you read the slices, not as a universal quality score.

Won't tell you · Latency, cost-per-token, and throughput — the index is intelligence-only. Use the AA API alongside for $/1M tokens.

187 score rows · 1 source · refreshed 2026-09-05

Use this when · Tool calling is the plumbing behind agents, automations, and research assistants. BFCL gives that plumbing a public test instead of treating every function-call demo as evidence.

Won't tell you · Whether the downstream tool result is useful — BFCL focuses on choosing and formatting calls, not on the whole product workflow after the call returns.

20 score rows · 1 source · refreshed 2026-06-09

CritPtCritical Physics Tasks

Use this when · When GPQA and HLE no longer tell you whether a model can do research-scale physics: original, unpublished problems with guess-resistant answers, still far from ceiling (frontier around 30%).

Won't tell you · Textbook or contest physics — problems are original research-scale tasks, not recycled qualifying-exam items.

12 score rows · 1 source · refreshed 2026-09-01

Use this when · When you need long-horizon, multi-language software work with behavioral verifiers — harder contamination posture than classic SWE-bench Verified.

Won't tell you · Python-only issue fixing like SWE-bench Verified — DeepSWE spans Go, Rust, TypeScript, and more.

8 score rows · 1 source · refreshed 2026-08-04

Use this when · When MATH and AIME no longer separate the frontier, this is the live math discriminator: original expert problems, Python-enabled, still far from ceiling.

Won't tell you · Tier 4 research-level problems — that 43-problem expansion is a separate, harder set. Do not mix Tiers 1-3 headlines with Tier 4.

15 score rows · 1 source · refreshed 2026-08-15

GDPvalReal-work deliverables vs experts

Use this when · MCQ suites miss whether a model can ship a real artifact. GDPval grades deliverables from 44 occupations against blind human experts — still a major Agents slice of AA Intelligence Index v4.2, now sharing that category with AA-Briefcase.

Won't tell you · VerdictPal endorsement — GDPval is one source, with its own task-mix choice.

6 score rows · 3 sources · refreshed 2026-06-22

GPQA DiamondGraduate-Level Google-Proof Q&A

Use this when · When you want graduate-level science that non-experts struggle to Google — a narrow but still discriminating knowledge/reasoning slice.

Won't tell you · General usefulness — it is a narrow slice of frontier science, not everyday tasks.

185 score rows · 2 sources · refreshed 2026-09-01

Use this when · Safety evidence is usually marketing copy or red-team anecdotes. HELM gives readers a public, repeatable place to inspect multiple safety metrics side by side.

Won't tell you · Real deployment risk by itself — a model can look safer in HELM than in a product with tools, memory, browsing, or weak policy enforcement.

9 score rows · 1 source · refreshed 2026-06-09

Humanity's Last ExamHLE

Use this when · When you need a hard, cross-domain expert exam — math through professional law and medicine — as a closed-book stress test, not a research-productivity proxy.

Won't tell you · Real research productivity — exam-style questions do not capture the messy search-and-synthesize work a researcher actually does.

180 score rows · 1 source · refreshed 2026-09-01

Use this when · Many classic tests are memorized, saturated, or stale. LiveBench earns a slot because it explicitly refreshes the question pool on a schedule.

Won't tell you · Static benchmark mastery — the point is freshness, so old snapshots should not be treated like permanent model truth.

21 score rows · 1 source · refreshed 2026-06-09

LiveCodeBench

Use this when · When you want contest-style coding with fresher problems that resist training-data leakage better than classic static suites.

Won't tell you · Multi-file refactors or IDE agent loops.

31 score rows · 1 source · refreshed 2026-09-01

Use this when · When you need a human preference signal at scale — how replies feel in blind side-by-side chat — not a correctness exam.

Won't tell you · Correctness — voters reward answers that look good, so formatting, length and confidence can beat accuracy.

13 score rows · 1 source · refreshed 2026-06-09

Use this when · Long context is now a product claim on almost every frontier model. LongBench v2 helps separate window size from actual long-context reasoning.

Won't tell you · Retrieval quality — the benchmark tests model use of provided context, not whether your search system found the right documents.

19 score rows · 1 source · refreshed 2026-06-09

Use this when · When you care whether a model can select, sequence, and recover tool calls over MCP — schema adherence and chain-of-tool reasoning, not chat polish.

Won't tell you · Harness quality — different MCP server implementations can artificially boost or suppress scores.

10 score rows · 1 source · refreshed 2026-06-09

MMLU-Pro

Use this when · When classic MMLU is too easy or too guessable — a stricter ten-option, reasoning-heavy knowledge exam across domains.

Won't tell you · Real-world open-ended ability — it is still a multiple-choice exam, just a stricter one.

39 score rows · 2 sources · refreshed 2026-09-01

MMMU-ProMMMU Pro

Use this when · Original MMMU no longer separates the frontier. MMMU-Pro is the live multimodal exam: shortcuts removed, ten options, still a spread below the old 88% ceiling.

Won't tell you · Original MMMU headlines — those still include text-solvable items. A 88%+ MMMU row is not an MMMU-Pro row.

8 score rows · 1 source · refreshed 2026-09-01

SciCode

Use this when · When you care about scientific programming with domain libraries and multi-step numerical code — not general SWE issue fixing.

Won't tell you · Wet-lab validation — generated code is graded against reference tests only.

180 score rows · 1 source · refreshed 2026-09-01

Terminal-BenchTerminal-Bench 2.1

Use this when · When you need realistic containerized terminal work — debug, files, commands, artifacts — as an agentic ops signal weighted in AA Index v4.1.

Won't tell you · The model alone — harness, tools, retries, time limits, and container setup can move scores dramatically.

184 score rows · 3 sources · refreshed 2026-09-01

Use this when · For source-heavy students and researchers this is the live science-agent discriminator: real lab-style workflows, still far from ceiling (best published 0.1 row is 30%).

Won't tell you · The model alone — every published 0.1 row names an agent harness (Claude Code, Codex, Grok Build). Harness choice moves the score.

9 score rows · 1 source · refreshed 2026-09-01

Use this when · When you want a practical, industry-flavoured composite — finance, coding, and related work — as one Vals evidence row, not a VerdictPal endorsement.

Won't tell you · Open reproducibility — parts of the benchmark use proprietary, non-public datasets, so readers cannot fully re-run the suite.

23 score rows · 1 source · refreshed 2026-09-05

Use this when · When you care about sustained business judgment over hundreds of simulated days — inventory, suppliers, pricing — not single-shot tool calls.

Won't tell you · Single-shot tool calls — runs exceed 20M output tokens; the test is sustained judgment.

10 score rows · 1 source · refreshed 2026-08-04

τ³-Bench Bankingtau3-bench-banking

Use this when · Replaces τ²-Bench Telecom in AA Index v4.1 with harder, more realistic agentic scenarios that better separate frontier models.

Won't tell you · Real bank integrations — simulated environment with controlled APIs, not production core banking systems.

106 score rows · 1 source · refreshed 2026-09-01