VerdictPal
Join

Benchmarks / Coding

SWE-bench Verified

Whether a model can resolve real GitHub issues from popular Python repos — it must produce a patch that makes the project's hidden test suite pass. 'Verified' is a 500-issue, human-validated subset of the original SWE-bench.

CodingFlagshipSaturatedMedium contamination riskSince 2024
What this does not measure
  • Anything but Python — the suite is Python-only, so it says little about other languages or front-end work.
  • Code quality or maintainability — a patch that passes tests can still be ugly or fragile.
  • The scaffolding matters as much as the model: agent harness, retries, and tool access swing scores by tens of points.
Analysis

Why this benchmark is useful

It turns code ability into a patch or pass-rate signal you can inspect. That makes it useful for comparing agent harnesses, not just base models.

Scope

Coverage map

Task family
Coding
Format
500 human-validated GitHub issues; success = the gold test suite passes after the model's patch.
Scoring
% of issues resolved (tests pass).
Maintainer
Princeton / OpenAI (Verified subset)
Reading guide

How to read the scores

Read SWE-bench Verified as a saturated signal with medium contamination risk. Compare models only when the source uses the same harness, prompting setup, sampling policy, and score unit.

Blind spots

What it does not cover

  • Anything but Python — the suite is Python-only, so it says little about other languages or front-end work.
  • Code quality or maintainability — a patch that passes tests can still be ugly or fragile.
  • The scaffolding matters as much as the model: agent harness, retries, and tool access swing scores by tens of points.
Scores

Evidence ledger

23 rows
2323 rows
93.9%best score
22source-checked
2sources
2026-06-29to 2026-05-30
SWE-bench22

official-json · scheduled

Vendor claims1

manual-snapshot · manual

Distribution

Where the rows land

0255075100

Normalized to this benchmark's axis (0–100). Open the table for raw units.

Timeline

Newest receipts

  1. LongCat-2.059.5% · Meituan LongCat-2.0 release materials
  2. Qwen 3.7 Max79.4% · SWE-bench Verified leaderboard
  3. MiniMax M378.4% · SWE-bench Verified leaderboard
  4. Grok 4.377.6% · SWE-bench Verified leaderboard
  5. Gemini 3.5 Flash77.2% · SWE-bench Verified leaderboard

Scores use this benchmark's own unit and axis, not a universal quality score.

#ModelRelease dateScoreProvenanceTrust
1Claude Mythos PreviewAnthropicBest-of-3 agent harness; compare to single-attempt runs on the same leaderboard before drawing conclusions.2026-06-0193.9%
SWE-bench official leaderboard2026-06-02
Source-checked
2Claude Opus 4.8Anthropic2026-05-2888.6%
SWE-bench official leaderboard2026-05-30
Source-checked
3Claude Opus 4.7 (Adaptive)Anthropic2026-04-1687.6%
SWE-bench official leaderboard2026-05-30
Source-checked
4GPT 5.5OpenAIDesk-attributed from GPT 5.5 card — reproducible harness score, not a vendor slide.2026-04-2382.5%
SWE-bench Verified leaderboard2026-06-01
Source-checked
4Claude Opus 4.5 (SEAL)Anthropic2025-11-0180.9%
SWE-bench SEAL leaderboard2026-05-30
Source-checked
5Claude Opus 4.6Anthropic2026-02-0180.8%
SWE-bench official leaderboard2026-05-30
Source-checked
6DeepSeek V4 Pro MaxDeepSeek2026-04-2480.6%
SWE-bench official leaderboard2026-05-30
Source-checked
6Gemini 3.1 ProGoogle2026-02-1980.6%
SWE-bench official leaderboard2026-05-30
Source-checked
8Kimi K2.6Moonshot2026-04-2080.2%
SWE-bench official leaderboard2026-05-30
Source-checked
8MiniMax M2.5MiniMax2026-04-0180.2%
SWE-bench official leaderboard2026-05-30
Source-checked
10GPT 5.2OpenAI2025-12-1180%
SWE-bench official leaderboard2026-05-30
Source-checked
10GPT 5.4OpenAI2026-03-0580%
SWE-bench official leaderboard2026-05-30
Source-checked
12Claude Sonnet 4.6Anthropic2026-02-1779.6%
SWE-bench official leaderboard2026-05-30
Source-checked
14Qwen 3.7 MaxAlibaba2026-05-2079.4%
SWE-bench Verified leaderboard2026-06-09
Source-checked
13DeepSeek V4 Flash MaxDeepSeek2026-05-1079%
SWE-bench official leaderboard2026-05-30
Source-checked
14Qwen 3.6 PlusAlibaba2026-04-0178.8%
SWE-bench official leaderboard2026-05-30
Source-checked
17MiniMax M3MiniMax2026-05-3178.4%
SWE-bench Verified leaderboard2026-06-09
Source-checked
15Gemini 3 FlashGoogle2025-12-1778%
SWE-bench official leaderboard2026-05-30
Source-checked
19Grok 4.3xAI2026-04-3077.6%
SWE-bench Verified leaderboard2026-06-09
Source-checked
20Gemini 3.5 FlashGoogle2026-05-1977.2%
SWE-bench Verified leaderboard2026-06-09
Source-checked
21Mistral Large 3Mistral2025-12-0276.2%
SWE-bench Verified leaderboard2026-06-09
Source-checked
22Llama 4 MaverickMeta2026-04-0574.8%
SWE-bench Verified leaderboard2026-06-09
Source-checked
23LongCat-2.0MeituanVendor-reported SWE-bench Pro score, cited as narrowly above GPT-5.5 in Meituan's release coverage. Not the canonical SWE-bench Verified row.2026-06-2959.5%
Meituan LongCat-2.0 release materials2026-06-29
Needs audit
Method

What it covers

Frontier models now cluster ~78–87% on Verified; contamination and harness variance pin discrimination. Prefer DeepSWE or SWE-Lancer for current coding-agent signal.

Score ceiling

Where it breaks down

No ceiling note is recorded yet. Treat clustering near the top as a warning that the benchmark may no longer separate frontier models.

Tools that report it

Receipts

Sources and further reading

← All benchmarksHow we score

The Pack · Editorial newsletter

New cards in your inbox. Free.

One short email when a card ships or changes status. No tracking, no third-party analytics. Unsubscribe in one click.