The public catalog of tools and models. It is a desk you can browse, not a ranking of winners.
Dossier · Editorial fit · Open the tool atlas
Link to this term with the heading.
tool page · model page
One finite page for one tool or model: dated pricing, privacy, failure modes, and a score you can trace. We never call these cards in reader copy.
Editorial fit · Failure mode · Evidence level
Link to this term with the heading.
0–100 · fit score · headline score
The 0–100 desk score on a tool dossier. It is a weighted judgment of research-workflow fit, not a user-star average and not a benchmark result.
Status labels · Evidence level · How the 0–100 lands
Link to this term with the heading.
Flagship · Solid · Lightweight · Draft
How finished a public page is — not how good the tool is. Flagship passed the full documentation gate. Solid is reliable with caveats. Lightweight is a first pass. Draft is scaffolding, not a recommendation. Flagship is never a silent synonym for best tool.
Editorial fit · Evidence level · Verification date · Benchmark status · Quality gate
Link to this term with the heading.
last checked · pricingChecked · privacyChecked
The date the desk last opened the vendor pricing page or privacy policy. It does not mean the whole dossier was re-scored that day.
Link to this term with the heading.
Benchmark pending · Benchmark ready
Whether a relevant first-party or sourced public test is attached. Pending is not a negative score — it means the desk has not published that evidence yet.
Evidence level · Editorial fit
Link to this term with the heading.
self-run · hands-on · synthesized · vendor claim
How close the desk got to the claim. Self-run means we ran the task. Hands-on means an editor used the tool. Synthesized compiles independent reports. Vendor claim is unverified marketing.
VerdictPal Lab · Source-checked · How we test
Link to this term with the heading.
editorial desk · Redaktion
The small Swiss student team that writes, checks dates, and signs off public pages. If the desk has not used a tool, the dossier stays labeled that way.
VerdictPal Lab · Evidence level · Who writes this
Link to this term with the heading.
newsletter · drop
The editorial newsletter. A drop ships when a tool, model, or method changes — never on a promised weekly cadence.
Link to this term with the heading.
pitfall · what breaks
A documented way the tool or workflow breaks in real student or research use. These stay visible on purpose, even when a vendor would rather hide them.
Link to this term with the heading.
blind spot · what it does not measure
The honest limit of a claim. On benchmark pages this is the list of things a high score does not prove.
Link to this term with the heading.
first-party · The Lab
Benchmarks we run ourselves, with a frozen protocol, question bank, and published method. Lab scores stay unpublished until a human grades and signs them off.
Citation fidelity · Evidence level · Open the Lab
Link to this term with the heading.
frontier
A current top-tier model, not last year's GPT-4-class leftover. Model compare on this site only lists frontier options.
Link to this term with the heading.
source-grounded · cited answer
Tying a claim to a source the reader can open. A fluent paragraph with no trail is not grounded, even if it sounds academic.
Citation fidelity · Hallucination
Link to this term with the heading.
cited claims
Whether a citation is faithful to its source: the source exists, the details are right, and it actually says what the sentence attributes to it. A reference can be real and still have low fidelity if the claim is not in the paper.
Grounding · Half-real citation · Hallucination
Link to this term with the heading.
made-up source · fabricated citation
A confident claim or citation that is not in the source. Fluency is not evidence. Check the PDF before you paste the footnote.
Link to this term with the heading.
hallucinated citation · fabricated reference · erfundene Quelle
A reference an AI tool writes for a source that does not exist, or that mixes real authors, titles and journals into a paper nobody published. It looks correctly formatted but cannot be found in Crossref, a library catalogue or Google Scholar. Check every AI-suggested reference before you cite it.
Ghost reference · Half-real citation · Hallucination
Link to this term with the heading.
phantom citation · Geisterreferenz · Phantomzitat
A citation to a work that does not exist. Libraries use this name; some people say phantom citation. The usual cause today is copying references from a chatbot without checking them. If no database, DOI or publisher page returns the title, treat it as a ghost.
Link to this term with the heading.
VerdictPal term
A reference to a paper that exists, but with wrong details: other authors, the wrong year, a wrong DOI or a claim the paper never makes. It survives a quick title search, so it is harder to catch than an invented one. Open the source and compare every field.
Hallucinated reference · Citation fidelity
Link to this term with the heading.
RAG answer · source-grounded answer · retrieval-grounded answer
An answer built from passages a tool first retrieved from a search index or your uploaded files, with citations to them. Retrieval makes real sources more likely but does not guarantee the claim matches the passage, so citation fidelity still needs checking.
Retrieval · Grounding · Citation fidelity
Link to this term with the heading.
open weights · open model
The model weights can be downloaded and run outside the vendor's chat box. Open-weight is not the same as open data, open training, or a privacy guarantee.
Link to this term with the heading.
eval · suite · test
A specific test of a specific skill: hard science questions, coding bugs, long documents, agent workflows. It is not a crown for best model.
Leaderboard · Family · Ledger · Benchmark library
Link to this term with the heading.
ranking · scoreboard
A single ordered list of models. VerdictPal refuses that frame. Different tests measure different jobs, so we never collapse Elo, pass@1, and accuracy into one fake ranking.
Link to this term with the heading.
score row · score database · evidence ledger
The dated table of sourced scores behind a benchmark page. One row is one model on one test, with a source URL and a date. Unlike metrics stay in their own columns.
Source-checked · Snapshot · Self-reported · Benchmark library
Link to this term with the heading.
verified · receipt
VerdictPal opened the cited primary source and confirmed the number. A sourced row without this stamp is still published, but treated with more caution.
Link to this term with the heading.
vendor score · announcement score
The figure comes from the model vendor's own announcement. Useful as a claim, weak as proof. Prefer an independent maintainer or a source-checked row.
Link to this term with the heading.
static snapshot · ingested
A frozen local copy of public scores. The live site does not scrape or call benchmark APIs while you read a page. Dates on the row tell you when that copy was taken.
Link to this term with the heading.
source date · ingested at
The paper trail on a score: who published it, the URL, the source date, and when VerdictPal ingested it. No number ships without that trail.
Link to this term with the heading.
official source
The group that owns the test: a paper team, Artificial Analysis, Vals, LMArena, or VerdictPal Lab. Official-maintainer rows beat a third-party re-run of the same suite.
Artificial Analysis · Vals AI · LMArena
Link to this term with the heading.
alias · model slug
The stable id VerdictPal uses for one model across sources. Vendors rename display strings; the key and its aliases keep GPT-5.5 from appearing as three different models.
Link to this term with the heading.
edition · ARC-AGI 2
A maintained edition of the same benchmark, such as ARC-AGI-1 versus ARC-AGI-2. Scores from different editions are not interchangeable.
Link to this term with the heading.
pass at 1 · pass@k
The share of tasks the model solves on its first try. pass@k would allow k attempts. Coding benches use this because one clean patch matters more than ten lucky retries.
Accuracy / percent · Harness · DeepSWE (coding primary)
Link to this term with the heading.
arena elo · rating
A relative rating from pairwise human votes, used by Chatbot Arena. 1200 versus 1300 is a preference gap, not 100 percent accuracy. Never line Elo up against a percent score.
Human preference · Leaderboard
Link to this term with the heading.
pct · % · score
Share of items answered correctly, usually 0–100. High accuracy on a saturated test can mean the suite is too easy, not that the model is done with the job.
Link to this term with the heading.
tps · output speed
How fast a model emits text once it starts answering. Useful for interactive work. It says nothing about whether the answer is right.
Time to first token · Token · Performance
Link to this term with the heading.
TTFT · latency
How long you wait before the first word appears. Separate from tokens-per-second: a model can start slowly and then stream quickly, or the reverse.
Tokens per second · Token · Performance
Link to this term with the heading.
expert ceiling · perfect score
The score a careful human or expert panel reaches on the same test. When models meet that line, the suite stops separating the frontier.
Link to this term with the heading.
gap to ceiling
The points still left between the best sourced model and the human ceiling. Small headroom means the test is nearly used up.
Link to this term with the heading.
intelligence index · Vals Index · AA Index
A weighted blend of several tests from one maintainer. Handy as a glance, opaque as a claim. VerdictPal does not publish the Vals or Artificial Analysis intelligence indexes as their own pages.
Artificial Analysis · Vals AI · Leaderboard
Link to this term with the heading.
no tools · no search
The model must answer from weights alone: no web, no calculator, no files. A closed-book score does not describe a student who is allowed to look things up.
Link to this term with the heading.
open-book · with tools · agent harness
The model may use a browser, code runner, or other tools during the test. Compare these rows only to other tool-augmented rows.
Closed-book · Agentic · Harness
Link to this term with the heading.
held-out set · hidden test
Some exams keep a hidden question set so models cannot memorize the public list. A score on only the public slice is easier to game.
Contamination · Accuracy / percent
Link to this term with the heading.
agent harness · eval harness
The wrapper that runs the model: prompts, tools, retries, time limits. The same model can look brilliant or clumsy under two harnesses. Read the harness before you trust a coding or agent score.
Agentic · pass@1 · Tool-augmented
Link to this term with the heading.
A number or capability statement that comes from the company selling the product. Published, labeled, and treated as unverified until the desk or an independent source checks it.
Self-reported · Evidence level
Link to this term with the heading.
AA · AA Index
An independent evaluator that re-runs public suites. VerdictPal stores a dated snapshot of the suites we still publish; we do not call AA while you load the page.
Composite index · Snapshot · Vals AI
Link to this term with the heading.
Vals Index
An independent score host. VerdictPal does not publish a Vals Index page. When an official maintainer and a third party both score a suite we keep, the official row wins.
Artificial Analysis · Maintainer
Link to this term with the heading.
Chatbot Arena · LMSYS
The public human-preference arena behind Elo scores. People vote on anonymous side-by-side answers. It measures taste, not citation quality.
Link to this term with the heading.
The chunk a model reads and writes — roughly a short word or part of a word. Prices are usually per million tokens. Context windows are counted in tokens, not pages.
Context window · Tokens per second
Link to this term with the heading.
context length
How much text the model can hold in one prompt, measured in tokens. A huge window does not guarantee the model still uses the sentence on page 40.
Link to this term with the heading.
RAG · search step
Finding the right passage before answering. Long-context tests put the passage in the prompt; search tools must fetch it first. Those are different skills.
Link to this term with the heading.
The test still separates today's frontier models. This is the set we would open first when you need live evidence.
Saturated · Solved / defunct · Primary benchmark
Link to this term with the heading.
used up
Top models cluster near the ceiling. The suite is still useful as history and method, but a new high score barely tells you who is ahead.
Active · Solved / defunct · Headroom
Link to this term with the heading.
defunct · ~100%
Verified top scores sit at or near 100 percent. Those suites are not on the atlas. This glossary is here so old papers still decode.
Saturated · Deprecated · Human ceiling
Link to this term with the heading.
retired
The maintainer stopped the suite, or we removed the public page. Old VerdictPal URLs redirect to the benchmark atlas.
Link to this term with the heading.
data leak · train/test leak · memorization
Risk that the test items leaked into training data. A high score on a contaminated suite can mean the model saw the answers, not that it learned the skill.
Public / private split · Saturated
Link to this term with the heading.
start here · discriminator
The one live test we would open first in a family. Math's primary is FrontierMath. Solved suites such as MATH and AIME are not published as pages.
Family · Active · Start-here strip
Link to this term with the heading.
task family
The job a benchmark claims to measure: knowledge, reasoning, math, coding, agents, and so on. Families keep unlike scores from being stacked into one chart.
Link to this term with the heading.
AA-Omniscience
Closed-book recall across school and professional subjects. A high knowledge score is not proof the model can run a literature review.
Link to this term with the heading.
HLE · GPQA
Hard, multi-step questions that should not be solvable by lookup alone. Still an exam, not a measure of messy research productivity.
Link to this term with the heading.
FrontierMath
Competition or research-level mathematics. Older public suites are largely solved; FrontierMath is the live discriminator on this site.
Primary benchmark · Solved / defunct
Link to this term with the heading.
DeepSWE · SciCode
Writing or repairing programs. Read the harness: a pass@1 on a hidden unit test is not the same as shipping a reviewed pull request.
Link to this term with the heading.
agent · Terminal-Bench
Multi-step work where the model must use tools, a terminal, or a workflow instead of answering one prompt. The harness is half the result.
Harness · Tool use · Tool-augmented
Link to this term with the heading.
MMMU · vision
Tasks that mix text with images or other media. A strong text-only model can still fail a figure, chart, or slide.
Link to this term with the heading.
AA-LCR · needle in a haystack
Using evidence that sits far apart inside one very long prompt. This is not the same as searching the live web.
Link to this term with the heading.
function calling · BFCL
Whether the model calls the right function, API, or browser action at the right time. Separate from writing good prose about the tool.
Link to this term with the heading.
speed · latency
Speed and responsiveness: output tokens per second, time to first token. These rows are not quality scores.
Tokens per second · Time to first token
Link to this term with the heading.
HELM Safety
Whether the model refuses harmful requests or leaks things it should not. A safety score is not a privacy policy and not a research-quality grade.
Link to this term with the heading.
arena · vibe check
Blind human votes on which answer feels better. Helpful for tone and usefulness. Silent on citations, contamination, and exam skill.
Link to this term with the heading.
Every public test on VerdictPal, in one alphabet. Open the dossier for scores, caveats, and sources.
Output tokens per second · Median output speed
Median output tokens per second after the first chunk arrives, measured on each model's default API provider with a ~1k-token input prompt (AA medium workload).
AA-LCR
Long-context synthesis and reasoning: 100 open-answer questions requiring models to integrate evidence across long inputs.
Critical Physics Tasks · Complex Research using Integrated Thinking - Physics Test
Unpublished research-level physics: 71 composite challenges (70 test + 1 example) written by 50+ active physicists across 11 subfields, plus 190 simpler checkpoint tasks.
Long-horizon software engineering on 113 contamination-free tasks across 91 repos and five languages — patches must pass hand-written behavioral verifiers, not string diffs.
FrontierMath Tiers 1-3 · FrontierMath v2
Hard unpublished mathematics: 295 private Tiers 1-3 problems (v2) where the model writes a Python answer() after iterative reasoning and code execution.
Graduate-Level Google-Proof Q&A
GPQA Diamond explained: graduate-level Google-proof Q&A in biology, physics, and chemistry. Items are written so non-experts fail even with web search; Diamond is the expert-validated hard subset (198 multiple-choice questions). A high score means correct answers on that set under the published protocol — not a general research-assistant grade.
HLE
Cross-domain expert-level questions — math, sciences, humanities, professional law and medicine — designed to be the hardest public exam a frontier model can take.
MMMU Pro · Massive Multi-discipline Multimodal Understanding Pro
Harder multimodal college exam: original MMMU items with text-only shortcuts filtered out, ten answer options instead of four, and a vision-only setting where the question is inside the image. 3,460 questions across 30 subjects.
AA-Omniscience Index · AA-Omniscience
Factual recall and knowledge calibration across 6,000 questions in 42 economically relevant topics and six domains. The score splits accuracy from a non-hallucination slice.
Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.
Terminal-Bench 2.1 · [email protected] · Terminal-Bench 4.0
Whether an agent can complete realistic, containerized terminal tasks (Terminal-Bench 2.1 on the public AA API): debugging code, handling files, running commands, and producing the artifact the hidden checker expects. Index v4.2 weighted this suite at 10%; Index v4.3 uses Terminal-Bench v4.0 in the composite instead.
TB-Science · Terminal-Bench-Science 0.1
Whether an agent can finish real scientific research workflows in a terminal: 70 expert-curated tasks across life, physical, Earth, mathematical, and engineering sciences, graded on concrete artifacts (analyses, simulations, proofs, code, data products).
The Pack · Editorial newsletter
One short email when a university changes its AI rule for theses, or a research tool changes its pricing or privacy terms. No tracking, no third-party analytics. Unsubscribe in one click.