Every benchmark, honestly read.

AA output speedOutput tokens per second

Median output tokens per second after the first chunk arrives, measured on each model's default API provider with a ~1k-token input prompt (AA medium workload).

Won't tell you · IDE or chat-app latency — only first-party API routes AA tracks as the default provider.

69 score rows · 1 source · refreshed 2026-07-21

AA time to first tokenTTFT

Median seconds until the first token returns on the model's default API provider with a ~1k-token input prompt.

Won't tell you · End-to-end task time for agent loops — TTFT is one hop, not a full coding session.

69 score rows · 1 source · refreshed 2026-07-21

Artificial Analysis Long Context ReasoningAA-LCR

Long-context synthesis and reasoning: 100 open-answer questions requiring models to integrate evidence across long inputs. 6% weight in AA Intelligence Index v4.1.

Won't tell you · Retrieval from external databases — the context is in-prompt only.

64 score rows · 1 source · refreshed 2026-07-21

Fluid, sample-efficient reasoning across the ARC Prize benchmark series — from static grid puzzles (ARC-AGI-1/2) to interactive agentic environments (ARC-AGI-3). Built to resist memorization and reward genuine generalization.

Won't tell you · Knowledge or language — tasks are deliberately knowledge-free, so scores say nothing about factual accuracy.

16 score rows · 1 source · refreshed 2026-06-22

Artificial Analysis Intelligence IndexAA Intelligence Index

AA Intelligence Index v4.1 — weighted composite emphasizing agentic workloads: GDPval-AA v2 (20%), Terminal-Bench 2.1 (16%), τ³-Bench Banking (14%), Humanity's Last Exam (12%), AA-Omniscience (12%), SciCode (8%), GPQA Diamond (6%), AA-LCR (6%), CritPt (6%). IFBench removed for saturation.

Won't tell you · Latency, cost-per-token, and throughput — the index is intelligence-only. Use the AA API alongside for $/1M tokens.

67 score rows · 1 source · refreshed 2026-07-21

Whether a model can choose and format function calls correctly across single-turn, multi-turn, parallel, and live API-style tool-use tasks.

Won't tell you · Whether the downstream tool result is useful — BFCL focuses on choosing and formatting calls, not on the whole product workflow after the call returns.

20 score rows · 1 source · refreshed 2026-06-09

Long-horizon software engineering on 113 contamination-free tasks across 91 repos and five languages — patches must pass hand-written behavioral verifiers, not string diffs.

Won't tell you · Python-only issue fixing like SWE-bench Verified — DeepSWE spans Go, Rust, TypeScript, and more.

6 score rows · 1 source · refreshed 2026-06-22

GDPvalReal-work deliverables vs experts

GDPval-AA v2 — real knowledge-work deliverables across 44 occupations, graded by blind pairwise comparison against human experts (Elo anchored at 1000). 20% weight in AA Intelligence Index v4.1.

Won't tell you · VerdictPal endorsement — GDPval is one source, with its own task-mix choice.

6 score rows · 3 sources · refreshed 2026-06-22

GPQA DiamondGraduate-Level Google-Proof Q&A

Hard graduate-level science questions (biology, physics, chemistry) written so that non-experts cannot solve them even with web access — the 'Diamond' subset is the highest-quality, expert-validated slice.

Won't tell you · General usefulness — it is a narrow slice of frontier science, not everyday tasks.

69 score rows · 2 sources · refreshed 2026-07-21

A multi-metric safety evaluation suite from Stanford CRFM that reports model behavior across safety-focused scenarios instead of collapsing everything into one headline score.

Won't tell you · Real deployment risk by itself — a model can look safer in HELM than in a product with tools, memory, browsing, or weak policy enforcement.

9 score rows · 1 source · refreshed 2026-06-09

Humanity's Last ExamHLE

Cross-domain expert-level questions — math, sciences, humanities, professional law and medicine — designed to be the hardest public exam a frontier model can take.

Won't tell you · Real research productivity — exam-style questions do not capture the messy search-and-synthesize work a researcher actually does.

65 score rows · 1 source · refreshed 2026-07-21

A contamination-limited benchmark that refreshes questions over time across reasoning, coding, math, data analysis, language, and instruction-following tasks.

Won't tell you · Static benchmark mastery — the point is freshness, so old snapshots should not be treated like permanent model truth.

21 score rows · 1 source · refreshed 2026-06-09

LiveCodeBench

Contamination-resistant coding problems drawn from recent contest releases — AA reruns models on the public harness.

Won't tell you · Multi-file refactors or IDE agent loops.

13 score rows · 1 source · refreshed 2026-07-21

Human preference at scale: people chat with two anonymous models side-by-side and vote for the better reply. Votes are turned into an Elo-style leaderboard.

Won't tell you · Correctness — voters reward answers that look good, so formatting, length and confidence can beat accuracy.

13 score rows · 1 source · refreshed 2026-06-09

Whether long-context models can reason over long, realistic inputs instead of merely accepting a large token window.

Won't tell you · Retrieval quality — the benchmark tests model use of provided context, not whether your search system found the right documents.

19 score rows · 1 source · refreshed 2026-06-09

How reliably a model selects, sequences, and recovers when invoking tools exposed via the Model Context Protocol — including schema adherence, error handling, and chain-of-tool reasoning.

Won't tell you · Harness quality — different MCP server implementations can artificially boost or suppress scores.

10 score rows · 1 source · refreshed 2026-06-09

MMLU-Pro

A harder MMLU successor: ten answer options instead of four, reasoning-heavy questions, and a scrubbed item set built to be less guessable.

Won't tell you · Real-world open-ended ability — it is still a multiple-choice exam, just a stricter one.

21 score rows · 2 sources · refreshed 2026-07-21

College-level questions that require reading images alongside text — charts, diagrams, chemical structures, medical scans — across 30 subjects.

Won't tell you · Pure vision quality — many items can be partly solved from the text alone, inflating scores.

8 score rows · 1 source · refreshed 2026-06-09

SciCode

Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.

Won't tell you · Wet-lab validation — generated code is graded against reference tests only.

65 score rows · 1 source · refreshed 2026-07-21

Terminal-BenchTerminal-Bench 2.1

Whether an agent can complete realistic, containerized terminal tasks (Terminal-Bench 2.1): debugging code, handling files, running commands, and producing the artifact the hidden checker expects. 16% weight in AA Intelligence Index v4.1.

Won't tell you · The model alone — harness, tools, retries, time limits, and container setup can move scores dramatically.

69 score rows · 3 sources · refreshed 2026-07-21

A Vals composite score across industry-flavoured tasks such as finance and coding, intended to approximate practical model usefulness on economically relevant work.

Won't tell you · Open reproducibility — parts of the benchmark use proprietary, non-public datasets, so readers cannot fully re-run the suite.

20 score rows · 1 source · refreshed 2026-07-19

Long-horizon business coherence: a model runs a simulated vending-machine company for 365 days — inventory, supplier email, pricing, refunds — scored on final bank balance (USD).

Won't tell you · Single-shot tool calls — runs exceed 20M output tokens; the test is sustained judgment.

5 score rows · 1 source · refreshed 2026-06-22