VerdictPal
Join

Benchmarks / Coding

SciCode

Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.

CodingSolidActiveMedium contaminationSince 2024
What this does not measure
  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.
Analysis

Why this benchmark is useful

When you care about scientific programming with domain libraries and multi-step numerical code — not general SWE issue fixing.

Scope

Coverage map

Task family
Coding
Format
Scientific coding problems with library dependencies.
Scoring
Percentage solved from AA runs.
Maintainer
SciCode authors
Reading guide

How to read the scores

Percentage solved from AA snapshot runs against reference tests. Wet-lab correctness and your local library versions are out of scope.

Blind spots

What it does not cover

  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.
Scores

Evidence ledger

180 rows
180180 rows
60.2%best score
180source-checked
1sources
2026-09-01source date
180

api · api

Distribution

Where the rows land

0255075100

Normalized to this benchmark's axis (0–100). Open the table for raw units.

Timeline

Newest receipts

  1. Claude Fable 560.2% ·
  2. Gemini 3.1 Pro Preview58.9% ·
  3. Kimi K358.7% ·
  4. Muse Spark 1.158.2% ·
  5. Gemini 3.7 Flash56.8% ·

Scores use this benchmark's own unit and axis, not a universal quality score.

#ModelRelease dateScoreProvenanceTrust
1Claude Fable 5Anthropic2026-06-0960.2%
2026-09-01
Source-checked
2Gemini 3.1 Pro PreviewGoogle2026-02-1958.9%
2026-09-01
Source-checked
3Kimi K3Moonshot2026-07-1658.7%
2026-09-01
Source-checked
4Muse Spark 1.1Meta2026-07-0958.2%
2026-09-01
Source-checked
5Gemini 3.7 FlashGoogle2026-08-1356.8%
2026-09-01
Source-checked
6GPT 5.4OpenAI2026-03-0556.6%
2026-09-01
Source-checked
7GLM-5.3Zhipu2026-08-1456.5%
2026-09-01
Source-checked
8Muse Spark 1.2Meta2026-08-0556.4%
2026-09-01
Source-checked
9Gemini 3 ProGoogle2025-11-1856.1%
2026-09-01
Source-checked
10GPT 5.5OpenAI2026-04-2356.1%
2026-09-01
Source-checked
11GPT-5.6 SolOpenAI2026-07-0956.1%
2026-09-01
Source-checked
12GPT-5.5 (high)OpenAI2026-04-2355.9%
2026-09-01
Source-checked
13Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic2026-07-2455.7%
2026-09-01
Source-checked
14Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)Anthropic2026-07-2455%
2026-09-01
Source-checked
15GPT-5.2 Codex (xhigh)OpenAI2025-12-1154.6%
2026-09-01
Source-checked
16Claude Opus 4.7Anthropic2026-04-1654.5%
2026-09-01
Source-checked
17Claude Opus 5 (Adaptive Reasoning, High Effort)Anthropic2026-07-2454.3%
2026-09-01
Source-checked
18Grok 4.5xAI2026-07-0854.1%
2026-09-01
Source-checked
19GPT-5.6 TerraOpenAI2026-07-0953.9%
2026-09-01
Source-checked
20Claude Sonnet 5Anthropic2026-06-3053.6%
2026-09-01
Source-checked
21Grok 4.6xAI2026-08-1253.6%
2026-09-01
Source-checked
22Claude Opus 4.8Anthropic2026-05-2853.5%
2026-09-01
Source-checked
23GPT-5.5 (medium)OpenAI2026-04-2353.5%
2026-09-01
Source-checked
24Kimi K2.6Moonshot2026-04-2053.5%
2026-09-01
Source-checked
25GPT-5.3 Codex (xhigh)OpenAI2026-02-0553.2%
2026-09-01
Source-checked
26Gemini 3.5 FlashGoogle2026-05-1953.1%
2026-09-01
Source-checked
27Gemini 3.5 Flash (medium)Google2026-05-1953%
2026-09-01
Source-checked
28Qwen3.8 MaxAlibaba2026-08-0352.9%
2026-09-01
Source-checked
29Gemini 3.6 Flash (high)Google2026-07-2152.7%
2026-09-01
Source-checked
30GPT-5.6 LunaOpenAI2026-07-0952.5%
2026-09-01
Source-checked
31GPT-5.2OpenAI2025-12-1152.1%
2026-09-01
Source-checked
32Claude Opus 4.6 (Adaptive Reasoning, Max Effort)Anthropic2026-02-0551.9%
2026-09-01
Source-checked
33GPT-5.5 (low)OpenAI2026-04-2351.6%
2026-09-01
Source-checked
34Qwen3.8 2.4T A95BAlibaba2026-08-1251.6%
2026-09-01
Source-checked
35Muse SparkOther2026-01-0151.5%
2026-09-01
Source-checked
36Kimi K3 (low)Moonshot2026-07-1651.2%
2026-09-01
Source-checked
37Claude Opus 5 (Adaptive Reasoning, Medium Effort)Anthropic2026-07-2450.7%
2026-09-01
Source-checked
38Gemini 3 Flash Preview (Reasoning)Google2025-12-1750.6%
2026-09-01
Source-checked
39GLM-5.2Zhipu2026-06-1350.5%
2026-09-01
Source-checked
40GPT-5.4 (low)OpenAI2026-03-0550.3%
2026-09-01
Source-checked
41GPT-5.5 Instant (May 2026)OpenAI2026-05-0550.3%
2026-09-01
Source-checked
42Grok Build 0.1 0616xAI2026-06-1650.2%
2026-09-01
Source-checked
43MiMo-V2.5-ProXiaomi2026-04-2250.2%
2026-09-01
Source-checked
44Claude Opus 4.7 (Non-reasoning, High Effort)Anthropic2026-04-1650.1%
2026-09-01
Source-checked
45DeepSeek V4 Flash 0731 (Reasoning, Max Effort)DeepSeek2026-07-3149.9%
2026-09-01
Source-checked
46Gemini 3 Flash PreviewGoogle2025-12-1749.9%
2026-09-01
Source-checked
47Gemini 3 Pro Preview (low)Google2025-11-1849.9%
2026-09-01
Source-checked
48GPT-5.4 MiniOpenAI2026-03-1749.9%
2026-09-01
Source-checked
49Claude Opus 4.5 (Reasoning)Anthropic2025-11-2449.5%
2026-09-01
Source-checked
50DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek2026-08-1349.2%
2026-09-01
Source-checked
51Kimi K2.5Moonshot2026-01-2749%
2026-09-01
Source-checked
52Gemini 3.5 Flash (minimal)Google2026-05-1948.8%
2026-09-01
Source-checked
53Qwen3.7 MaxAlibaba2026-05-1948.8%
2026-09-01
Source-checked
54Inkling SmallOther2026-07-3048.7%
2026-09-01
Source-checked
55GPT-5.5 Instant (June 2026)OpenAI2026-06-2548.6%
2026-09-01
Source-checked
56Claude Opus 5 (Adaptive Reasoning, Low Effort)Anthropic2026-07-2448%
2026-09-01
Source-checked
57Agnes 2.5 Pro BetaOther2026-08-2647.6%
2026-09-01
Source-checked
58Hy3Other2026-07-0647.6%
2026-09-01
Source-checked
59Kimi K2.7 CodeMoonshot2026-06-1247.5%
2026-09-01
Source-checked
60GPT-5.5 (Non-reasoning)OpenAI2026-04-2347.3%
2026-09-01
Source-checked
61Grok 4.3xAI2026-04-3047.3%
2026-09-01
Source-checked
62Claude Opus 4.5Anthropic2025-11-2447%
2026-09-01
Source-checked
63MiniMax M2.7MiniMax2026-04-1547%
2026-09-01
Source-checked
64Claude Sonnet 4.6Anthropic2026-02-1746.9%
2026-09-01
Source-checked
65GPT-5.4 NanoOpenAI2026-03-1746.9%
2026-09-01
Source-checked
66Qwen3.6 Max PreviewAlibaba2026-04-2046.9%
2026-09-01
Source-checked
67Qwen3.8-Flash-NextAlibaba2026-08-2646.9%
2026-09-01
Source-checked
68Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)Anthropic2026-02-1746.8%
2026-09-01
Source-checked
69DeepSeek V4 Flash Vision (Reasoning, Max Effort)DeepSeek2026-08-2146.6%
2026-09-01
Source-checked
70DeepSeek V4 Pro (Reasoning, High Effort)DeepSeek2026-04-2446.4%
2026-09-01
Source-checked
71GLM-5 (Reasoning)Zhipu2026-02-1146.2%
2026-09-01
Source-checked
72GPT-5.2 (medium)OpenAI2025-12-1146.2%
2026-09-01
Source-checked
73GLM-5.3-FlashZhipu2026-08-2646.1%
2026-09-01
Source-checked
74Inkling (xhigh)Other2026-07-1546.1%
2026-09-01
Source-checked
75Claude Opus 4.6Anthropic2026-02-0545.7%
2026-09-01
Source-checked
76Grok 4xAI2025-07-1045.7%
2026-09-01
Source-checked
77Grok 4.20 ReasoningxAI2026-03-0545.6%
2026-09-01
Source-checked
78Solar Open2 250BOther2026-08-1245.6%
2026-09-01
Source-checked
79Qwen 3.7 PlusAlibaba2026-06-0245.5%
2026-09-01
Source-checked
80MiniMax M3MiniMax2026-05-3145.4%
2026-09-01
Source-checked
81GLM-4.7 (Reasoning)Zhipu2025-12-2245.1%
2026-09-01
Source-checked
82Claude 4.5 Sonnet (Reasoning)Anthropic2025-09-2944.7%
2026-09-01
Source-checked
83Grok 4.20 0309 (Reasoning)xAI2026-03-1044.7%
2026-09-01
Source-checked
84Qwen3.8 27BAlibaba2026-08-1444.7%
2026-09-01
Source-checked
85Grok 4.3 (medium)xAI2026-04-3044.6%
2026-09-01
Source-checked
86Motif 3 (Beta)Other2026-07-2144.3%
2026-09-01
Source-checked
87Claude Sonnet 4.6 (Non-reasoning, Low Effort)Anthropic2026-02-1744.1%
2026-09-01
Source-checked
88GLM-5.1Zhipu2026-04-0743.8%
2026-09-01
Source-checked
89GLM-5-TurboZhipu2026-03-1543.6%
2026-09-01
Source-checked
90Muse GlimmerMeta2026-08-1043.6%
2026-09-01
Source-checked
91Solar Pro 4Other2026-08-0643.6%
2026-09-01
Source-checked
92GLM 5V Turbo (Reasoning)Zhipu2026-04-0143.5%
2026-09-01
Source-checked
93Gemma 4 31B ITGoogle2026-03-0143.4%
2026-09-01
Source-checked
94GPT-5.1OpenAI2025-11-1343.3%
2026-09-01
Source-checked
95MiMo-V2.5Xiaomi2026-04-2243.1%
2026-09-01
Source-checked
96Qwen3 Max ThinkingAlibaba2026-01-2643.1%
2026-09-01
Source-checked
97Apodex 1.1Other2026-08-3042.9%
2026-09-01
Source-checked
98GPT-5 (high)OpenAI2025-08-0742.9%
2026-09-01
Source-checked
99MiniMax M2.5MiniMax2026-04-0142.6%
2026-09-01
Source-checked
100MiMo-V2-ProXiaomi2026-03-1842.5%
2026-09-01
Source-checked
101DeepSeek V4 Pro (Non-reasoning)DeepSeek2026-04-2442.4%
2026-09-01
Source-checked
102Kimi K2 ThinkingMoonshot2025-11-0642.4%
2026-09-01
Source-checked
103Ring-2.6-1TOther2026-05-0842.4%
2026-09-01
Source-checked
104Agnes 2.5 Pro AlphaOther2026-07-2442.2%
2026-09-01
Source-checked
105DeepSeek V4 Flash (Reasoning, High Effort)DeepSeek2026-04-2442%
2026-09-01
Source-checked
106Qwen3.5 122B A10B (Reasoning)Alibaba2026-02-2442%
2026-09-01
Source-checked
107Qwen3.5 397B A17B (Reasoning)Alibaba2026-02-1642%
2026-09-01
Source-checked
108Gemini 3.1 Flash LiteGoogle2026-05-0741.9%
2026-09-01
Source-checked
109Grok 4.3 (low)xAI2026-04-3041.9%
2026-09-01
Source-checked
110Nex-N2-ProOther2026-06-0241.8%
2026-09-01
Source-checked
111Hy3-preview (Reasoning)Other2026-04-2341.2%
2026-09-01
Source-checked
112GPT-5 (medium)OpenAI2025-08-0741.1%
2026-09-01
Source-checked
113Ling 3.0 FlashOther2026-08-0441.1%
2026-09-01
Source-checked
114Qwen3.5 397B A17B (Non-reasoning)Alibaba2026-02-1641.1%
2026-09-01
Source-checked
115K-EXAONE 2.0 0803Other2026-08-1241%
2026-09-01
Source-checked
116OpenAI o3OpenAI2025-04-1641%
2026-09-01
Source-checked
117Claude 4.1 Opus (Reasoning)Anthropic2025-08-0540.9%
2026-09-01
Source-checked
118Gemini 3.5 Flash-LiteGoogle2026-07-2140.9%
2026-09-01
Source-checked
119GPT-5 Codex (high)OpenAI2025-09-2340.9%
2026-09-01
Source-checked
120MiniMax-M2.1MiniMax2025-12-2340.7%
2026-09-01
Source-checked
121Qwen3.6 PlusAlibaba2026-04-0240.7%
2026-09-01
Source-checked
122Motif 3Other2026-08-1240.6%
2026-09-01
Source-checked
123DeepSeek R1DeepSeek2025-01-2040.3%
2026-09-01
Source-checked
124GPT-5.1 Codex (high)OpenAI2025-11-1340.2%
2026-09-01
Source-checked
125Step 3.7 FlashStepFun2026-06-0140%
2026-09-01
Source-checked
126Nemotron 3 Ultra 550B A55B (Reasoning)NVIDIA2026-06-0439.9%
2026-09-01
Source-checked
127Claude 4 Opus (Reasoning)Anthropic2025-05-2239.8%
2026-09-01
Source-checked
128Qwen3.6 27B (Reasoning)Alibaba2026-04-2239.8%
2026-09-01
Source-checked
129Mistral Medium 3.5Mistral2026-04-2939.6%
2026-09-01
Source-checked
130Kimi K2.6 (Non-reasoning)Other2026-04-2039.5%
2026-09-01
Source-checked
131MiMo-V2-Omni-0327Xiaomi2026-03-2739.5%
2026-09-01
Source-checked
132Qwen3.5 27B (Reasoning)Alibaba2026-02-2439.5%
2026-09-01
Source-checked
133Hy3-preview (Non-reasoning)Other2026-04-2339.4%
2026-09-01
Source-checked
134MiMo-V2-Flash (Reasoning)Xiaomi2025-12-1639.4%
2026-09-01
Source-checked
135GPT-5 (low)OpenAI2025-08-0739.1%
2026-09-01
Source-checked
136MiMo-V2.5-Pro (Non-reasoning)Xiaomi2026-04-2239.1%
2026-09-01
Source-checked
137DeepSeek V3.2 (Reasoning)DeepSeek2025-12-0138.9%
2026-09-01
Source-checked
138A.X-K2Other2026-08-1238.5%
2026-09-01
Source-checked
139GLM-5 (Non-reasoning)Zhipu2026-02-1138.3%
2026-09-01
Source-checked
140KAT Coder Pro V2Other2026-03-2738.3%
2026-09-01
Source-checked
141MiMo-V2-Flash (Feb 2026)Xiaomi2025-12-1638.3%
2026-09-01
Source-checked
142Gemma 4 12B (Reasoning)Google2026-06-0738.2%
2026-09-01
Source-checked
143JT-4.1 Flash 236B A21BOther2026-07-0938.2%
2026-09-01
Source-checked
144North Mini CodeCohere2026-06-0938.2%
2026-09-01
Source-checked
145Command ACohere2026-03-1537.8%
2026-09-01
Source-checked
146Grok 4.3 (Non-reasoning)xAI2026-04-3037.4%
2026-09-01
Source-checked
147DeepSeek V4 Flash (Non-reasoning)DeepSeek2026-04-2437.3%
2026-09-01
Source-checked
148Qwen3.6 27B (Non-reasoning)Alibaba2026-04-2237.3%
2026-09-01
Source-checked
149Ling-2.6-1TOther2026-04-2337%
2026-09-01
Source-checked
150MiMo-V2-OmniXiaomi2026-03-1936.7%
2026-09-01
Source-checked
151Granite 4.2 30BOther2026-08-2536.6%
2026-09-01
Source-checked
152Mistral Large 3Mistral2025-12-0236.2%
2026-09-01
Source-checked
153GLM-5.2 (Non-reasoning)Zhipu2026-06-1636.1%
2026-09-01
Source-checked
154OpenAI o1OpenAI2024-09-1235.8%
2026-09-01
Source-checked
155Qwen3.6 35B A3B (Reasoning)Alibaba2026-04-1635.8%
2026-09-01
Source-checked
156LongCat-2.0Meituan2026-06-2935.4%
2026-09-01
Source-checked
157DiffusionGemma 26B A4BGoogle2026-06-1034.3%
2026-09-01
Source-checked
158G9v3-39A5BOther2026-08-0334%
2026-09-01
Source-checked
159Llama 4 MaverickMeta2026-04-0533.1%
2026-09-01
Source-checked
160HyperNova 60B 2605Multiverse2026-05-0633%
2026-09-01
Source-checked
161Grok 4.20 0309 v2 (Non-reasoning)xAI2026-04-0732.8%
2026-09-01
Source-checked
162Nemotron 3.5 LightningNVIDIA2026-08-1131.6%
2026-09-01
Source-checked
163Granite 4.2 8BOther2026-08-2530.4%
2026-09-01
Source-checked
164Gemma 4 12B (Non-reasoning)Google2026-06-1029.7%
2026-09-01
Source-checked
165JT-35B-FlashOther2026-05-1429.1%
2026-09-01
Source-checked
166EXAONE 4.5 33BOther2026-04-0928%
2026-09-01
Source-checked
167Nemotron 3 Nano Omni 30B A3B ReasoningNVIDIA2026-04-2927.8%
2026-09-01
Source-checked
168JT-MINIOther2026-04-1527.2%
2026-09-01
Source-checked
169Ling 2.6 FlashOther2026-04-2127.1%
2026-09-01
Source-checked
170Granite 4.1 30BOther2026-04-2925.8%
2026-09-01
Source-checked
171Granite 4.2 3BOther2026-08-2524.9%
2026-09-01
Source-checked
172Ling 3.0 TinyOther2026-08-0624.2%
2026-09-01
Source-checked
173Claude 3 OpusAnthropic2024-03-0423.3%
2026-09-01
Source-checked
174G9v3-3BOther2026-07-2317.7%
2026-09-01
Source-checked
175Llama 4 ScoutMeta2026-04-0517%
2026-09-01
Source-checked
176LFM2.5-8B-A1BLiquid2026-06-087.8%
2026-09-01
Source-checked
177MiniCPM5-1B (Reasoning)OpenBMB2026-06-044.4%
2026-09-01
Source-checked
178MiniCPM-V 4.6 1.3BOpenBMB2026-05-112.1%
2026-09-01
Source-checked
179MiniCPM5-1B (Non-reasoning)OpenBMB2026-05-251.4%
2026-09-01
Source-checked
180Qwen3.6 35B A3B (Non-reasoning)Alibaba2026-04-161.3%
2026-09-01
Source-checked
Method

What it covers

This benchmark sits in the coding family. It uses Scientific coding problems with library dependencies. The score should travel with its task format, scoring method, source date, and benchmark version.

Score ceiling

Where it breaks down

No ceiling note is recorded yet. Treat clustering near the top as a warning that the benchmark may no longer separate frontier models.

Receipts

Sources and further reading

← All benchmarksHow we score

Questions about this benchmark

Every answer below is assembled from the dated fields on this page. Nothing is written separately for search.

What does SciCode measure?

Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.

What does a high SciCode score not prove?

A strong SciCode result says nothing about:

  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.

How is SciCode scored?

Percentage solved from AA runs. Task format: Scientific coding problems with library dependencies.

Is SciCode saturated?

SciCode is currently marked Active in the atlas.

Can SciCode results be contaminated by training data?

Contamination risk for SciCode is graded Medium contamination. Treat every row on this page as a public claim with a source and a date, not as a controlled experiment.

The Pack · Editorial newsletter

New tools in your inbox. Free.

One short email when a tool ships or changes status. No tracking, no third-party analytics. Unsubscribe in one click.