Why this benchmark is useful
Editorial brief pendingWe publish the methodology and ledger first; benchmark-specific analysis ships after desk review.
Benchmarks / Coding
SciCode
Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.
CodingSolidActiveMedium contamination riskSince 2024
What this does not measure
- Wet-lab validation — generated code is graded against reference tests only.
- Library versions in your environment — harness pins dependency sets.
Coverage map
- Task family
- Coding
- Format
- Scientific coding problems with library dependencies.
- Scoring
- Percentage solved from AA runs.
- Maintainer
- SciCode authors
How to read the scores
Reading guide pending. Use task format, scoring method, and source dates in the ledger until the desk brief ships.
What it does not cover
- Wet-lab validation — generated code is graded against reference tests only.
- Library versions in your environment — harness pins dependency sets.
Scores
65 rowsEvidence ledger
6565 rows
60.2%best score
65source-checked
1sources
2026-07-21source date
Where the rows land
0255075100
Normalized to this benchmark's axis (0–100). Open the table for raw units.
Newest receipts
- Claude Fable 560.2% ·
- Gemini 3.1 Pro Preview58.9% ·
- Kimi K358.7% ·
- Muse Spark 1.158.2% ·
- GPT 5.456.6% ·
Scores use this benchmark's own unit and axis, not a universal quality score.
| # | Model | Release date | Score | Provenance | Trust |
|---|---|---|---|---|---|
| 1 | Claude Fable 5Anthropic | 2026-06-09 | 60.2% | 2026-07-21 | Source-checked |
| 2 | Gemini 3.1 Pro PreviewGoogle | 2026-02-19 | 58.9% | 2026-07-21 | Source-checked |
| 3 | Kimi K3Moonshot | 2026-07-16 | 58.7% | 2026-07-21 | Source-checked |
| 4 | Muse Spark 1.1Meta | 2026-07-09 | 58.2% | 2026-07-21 | Source-checked |
| 5 | GPT 5.4OpenAI | 2026-03-05 | 56.6% | 2026-07-21 | Source-checked |
| 6 | Gemini 3 ProGoogle | 2025-11-18 | 56.1% | 2026-07-21 | Source-checked |
| 7 | GPT 5.5OpenAI | 2026-04-23 | 56.1% | 2026-07-21 | Source-checked |
| 8 | GPT-5.6 SolOpenAI | 2026-07-09 | 56.1% | 2026-07-21 | Source-checked |
| 9 | GPT-5.5 (high)OpenAI | 2026-04-23 | 55.9% | 2026-07-21 | Source-checked |
| 10 | Claude Opus 4.7Anthropic | 2026-04-16 | 54.5% | 2026-07-21 | Source-checked |
| 11 | Grok 4.5xAI | 2026-07-08 | 54.1% | 2026-07-21 | Source-checked |
| 12 | GPT-5.6 TerraOpenAI | 2026-07-09 | 53.9% | 2026-07-21 | Source-checked |
| 13 | Claude Sonnet 5Anthropic | 2026-06-30 | 53.6% | 2026-07-21 | Source-checked |
| 14 | Claude Opus 4.8Anthropic | 2026-05-28 | 53.5% | 2026-07-21 | Source-checked |
| 15 | GPT-5.5 (medium)OpenAI | 2026-04-23 | 53.5% | 2026-07-21 | Source-checked |
| 16 | Kimi K2.6Moonshot | 2026-04-20 | 53.5% | 2026-07-21 | Source-checked |
| 17 | GPT-5.3 Codex (xhigh)OpenAI | 2026-02-05 | 53.2% | 2026-07-21 | Source-checked |
| 18 | Gemini 3.5 FlashGoogle | 2026-05-19 | 53.1% | 2026-07-21 | Source-checked |
| 19 | Gemini 3.5 Flash (medium)Google | 2026-05-19 | 53% | 2026-07-21 | Source-checked |
| 20 | GPT-5.6 LunaOpenAI | 2026-07-09 | 52.5% | 2026-07-21 | Source-checked |
| 21 | GPT-5.2OpenAI | 2025-12-11 | 52.1% | 2026-07-21 | Source-checked |
| 22 | Claude Opus 4.6 (Adaptive Reasoning, Max Effort)Anthropic | 2026-02-05 | 51.9% | 2026-07-21 | Source-checked |
| 23 | GPT-5.5 (low)OpenAI | 2026-04-23 | 51.6% | 2026-07-21 | Source-checked |
| 24 | Muse SparkOther | 2026-01-01 | 51.5% | 2026-07-21 | Source-checked |
| 25 | GLM-5.2Zhipu | 2026-06-13 | 50.5% | 2026-07-21 | Source-checked |
| 26 | MiMo-V2.5-ProXiaomi | 2026-04-22 | 50.2% | 2026-07-21 | Source-checked |
| 27 | Claude Opus 4.7 (Non-reasoning, High Effort)Anthropic | 2026-04-16 | 50.1% | 2026-07-21 | Source-checked |
| 28 | DeepSeek V4 Pro (Reasoning, Max Effort)DeepSeek | 2026-04-24 | 50% | 2026-07-21 | Source-checked |
| 29 | Gemini 3 Flash PreviewGoogle | 2025-12-17 | 49.9% | 2026-07-21 | Source-checked |
| 30 | GPT-5.4 MiniOpenAI | 2026-03-17 | 49.9% | 2026-07-21 | Source-checked |
| 31 | Kimi K2.5Moonshot | 2026-01-27 | 49% | 2026-07-21 | Source-checked |
| 32 | Qwen3.7 MaxAlibaba | 2026-05-19 | 48.8% | 2026-07-21 | Source-checked |
| 33 | Kimi K2.7 CodeMoonshot | 2026-06-12 | 47.5% | 2026-07-21 | Source-checked |
| 34 | Grok 4.3xAI | 2026-04-30 | 47.3% | 2026-07-21 | Source-checked |
| 35 | Claude Opus 4.5Anthropic | 2025-11-24 | 47% | 2026-07-21 | Source-checked |
| 36 | MiniMax M2.7MiniMax | 2026-04-15 | 47% | 2026-07-21 | Source-checked |
| 37 | Claude Sonnet 4.6Anthropic | 2026-02-17 | 46.9% | 2026-07-21 | Source-checked |
| 38 | GPT-5.4 NanoOpenAI | 2026-03-17 | 46.9% | 2026-07-21 | Source-checked |
| 39 | Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)Anthropic | 2026-02-17 | 46.8% | 2026-07-21 | Source-checked |
| 40 | Claude Opus 4.6Anthropic | 2026-02-05 | 45.7% | 2026-07-21 | Source-checked |
| 41 | Grok 4.20 ReasoningxAI | 2026-03-05 | 45.6% | 2026-07-21 | Source-checked |
| 42 | Qwen 3.7 PlusAlibaba | 2026-06-02 | 45.5% | 2026-07-21 | Source-checked |
| 43 | MiniMax M3MiniMax | 2026-05-31 | 45.4% | 2026-07-21 | Source-checked |
| 44 | GLM-5.1Zhipu | 2026-04-07 | 43.8% | 2026-07-21 | Source-checked |
| 45 | Gemma 4 31B ITGoogle | 2026-03-01 | 43.4% | 2026-07-21 | Source-checked |
| 46 | GPT-5.1OpenAI | 2025-11-13 | 43.3% | 2026-07-21 | Source-checked |
| 47 | MiniMax M2.5MiniMax | 2026-04-01 | 42.6% | 2026-07-21 | Source-checked |
| 48 | Gemini 3.1 Flash LiteGoogle | 2026-05-07 | 41.9% | 2026-07-21 | Source-checked |
| 49 | OpenAI o3OpenAI | 2025-04-16 | 41% | 2026-07-21 | Source-checked |
| 50 | DeepSeek R1DeepSeek | 2025-01-20 | 40.3% | 2026-07-21 | Source-checked |
| 51 | Step 3.7 FlashStepFun | 2026-06-01 | 40% | 2026-07-21 | Source-checked |
| 52 | Nemotron 3 Ultra 550B A55B (Reasoning)NVIDIA | 2026-06-04 | 39.9% | 2026-07-21 | Source-checked |
| 53 | Gemma 4 12B (Reasoning)Google | 2026-06-07 | 38.2% | 2026-07-21 | Source-checked |
| 54 | North Mini CodeCohere | 2026-06-09 | 38.2% | 2026-07-21 | Source-checked |
| 55 | Mistral Large 3Mistral | 2025-12-02 | 36.2% | 2026-07-21 | Source-checked |
| 56 | OpenAI o1OpenAI | 2024-09-12 | 35.8% | 2026-07-21 | Source-checked |
| 57 | LongCat-2.0Meituan | 2026-06-29 | 35.4% | 2026-07-21 | Source-checked |
| 58 | Llama 4 MaverickMeta | 2026-04-05 | 33.1% | 2026-07-21 | Source-checked |
| 59 | HyperNova 60B 2605Multiverse | 2026-05-06 | 33% | 2026-07-21 | Source-checked |
| 60 | Gemma 4 12B (Non-reasoning)Google | 2026-06-10 | 29.7% | 2026-07-21 | Source-checked |
| 61 | Command ACohere | 2026-03-15 | 28.1% | 2026-07-21 | Source-checked |
| 62 | Claude 3 OpusAnthropic | 2024-03-04 | 23.3% | 2026-07-21 | Source-checked |
| 63 | Llama 4 ScoutMeta | 2026-04-05 | 17% | 2026-07-21 | Source-checked |
| 64 | LFM2.5-8B-A1BLiquid | 2026-06-08 | 7.8% | 2026-07-21 | Source-checked |
| 65 | MiniCPM5-1B (Reasoning)OpenBMB | 2026-06-04 | 4.4% | 2026-07-21 | Source-checked |
What it covers
Data-quality note pending. Every ledger row still carries source URL, source date, and ingest timestamp.
Where it breaks down
No ceiling note is recorded yet. Treat clustering near the top as a warning that the benchmark may no longer separate frontier models.
Receipts
Sources and further reading
- AA SciCode· checked 2026-06-10
The Pack · Editorial newsletter
New cards in your inbox. Free.
One short email when a card ships or changes status. No tracking, no third-party analytics. Unsubscribe in one click.