VerdictPal
Join

Benchmarks / Coding

SciCode

Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.

CodingSolidActiveMedium contaminationSince 2024
What this does not measure
  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.
Analysis

Why this benchmark is useful

When you care about scientific programming with domain libraries and multi-step numerical code — not general SWE issue fixing.

Scope

Coverage map

Task family
Coding
Format
Scientific coding problems with library dependencies.
Scoring
Percentage solved from AA runs.
Maintainer
SciCode authors
Reading guide

How to read the scores

Percentage solved from AA snapshot runs against reference tests. Wet-lab correctness and your local library versions are out of scope.

Blind spots

What it does not cover

  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.
Scores

Evidence ledger

97 rows
9797 rows
66.9%best score
97source-checked
1sources
2026-09-28source date
97

api · api

Distribution

Where the rows land

0255075100

Normalized to this benchmark's axis (0–100). Open the table for raw units.

Timeline

Newest receipts

  1. Claude Opus 5.566.9% ·
  2. Claude Fable 5.163.1% ·
  3. Claude Fable 561% ·
  4. MiMo-V2.6-Pro60.9% ·
  5. Kimi K359.5% ·

Scores use this benchmark's own unit and axis, not a universal quality score.

#ModelRelease dateScoreProvenanceTrust
1Claude Opus 5.5Anthropic2026-09-2266.9%
2026-09-28
Source-checked
2Claude Fable 5.1Anthropic2026-09-0163.1%
2026-09-28
Source-checked
3Claude Fable 5Anthropic2026-06-0961%
2026-09-28
Source-checked
4MiMo-V2.6-ProXiaomi2026-09-2260.9%
2026-09-28
Source-checked
5Kimi K3Moonshot2026-07-1659.5%
2026-09-28
Source-checked
6GLM-5.3Zhipu2026-08-1459%
2026-09-28
Source-checked
7Muse Spark 1.1Meta2026-07-0958.8%
2026-09-28
Source-checked
8Muse Spark 1.3Meta2026-09-0258.8%
2026-09-28
Source-checked
9Gemini 3.1 Pro PreviewGoogle2026-02-1958.7%
2026-09-28
Source-checked
10GPT-6 SolOpenAI2026-09-2257.6%
2026-09-28
Source-checked
11Grok 4.7xAI2026-09-2157.4%
2026-09-28
Source-checked
12Muse Spark 1.2Meta2026-08-0557.4%
2026-09-28
Source-checked
13Gemini 3.7 FlashGoogle2026-08-1357.2%
2026-09-28
Source-checked
14GPT-5.6 SolOpenAI2026-07-0957.1%
2026-09-28
Source-checked
15Gemini 3.8 FlashGoogle2026-09-0256.6%
2026-09-28
Source-checked
16GPT-6 AstraOpenAI2026-09-0356.5%
2026-09-28
Source-checked
17Grok 4.6xAI2026-08-1256.5%
2026-09-28
Source-checked
18Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic2026-07-2456.4%
2026-09-28
Source-checked
19GPT-5.5 (high)OpenAI2026-04-2356.1%
2026-09-28
Source-checked
20GPT 5.5OpenAI2026-04-2355.8%
2026-09-28
Source-checked
21Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)Anthropic2026-07-2455.7%
2026-09-28
Source-checked
22Claude Opus 5 (Adaptive Reasoning, High Effort)Anthropic2026-07-2455.4%
2026-09-28
Source-checked
23GPT-5.6 TerraOpenAI2026-07-0955%
2026-09-28
Source-checked
24Grok 4.5xAI2026-07-0855%
2026-09-28
Source-checked
25GPT-6 LunaOpenAI2026-09-2254.6%
2026-09-28
Source-checked
26GPT-5.5 (medium)OpenAI2026-04-2354.5%
2026-09-28
Source-checked
27Claude Opus 4.8Anthropic2026-05-2854.4%
2026-09-28
Source-checked
28Claude Sonnet 5Anthropic2026-06-3054.3%
2026-09-28
Source-checked
29Qwen3.8 2.4T A95BAlibaba2026-08-1254.1%
2026-09-28
Source-checked
30Gemini 3.5 FlashGoogle2026-05-1953.9%
2026-09-28
Source-checked
31GPT-5.6 LunaOpenAI2026-07-0953.6%
2026-09-28
Source-checked
32Gemini 3.6 Flash (high)Google2026-07-2153.4%
2026-09-28
Source-checked
33Kimi K3 (low)Moonshot2026-07-1652.7%
2026-09-28
Source-checked
34GPT-5.5 Instant (June 2026)OpenAI2026-06-2552.5%
2026-09-28
Source-checked
35GPT-5.4 MiniOpenAI2026-03-1752.1%
2026-09-28
Source-checked
36Qwen3.8 MaxAlibaba2026-08-0352.1%
2026-09-28
Source-checked
37DeepSeek V4.1 FlashDeepSeek2026-09-1051.9%
2026-09-28
Source-checked
38GLM-5.3-FlashZhipu2026-08-2651.6%
2026-09-28
Source-checked
39Claude Opus 5 (Adaptive Reasoning, Medium Effort)Anthropic2026-07-2451.5%
2026-09-28
Source-checked
40Kimi K2.6Moonshot2026-04-2051.5%
2026-09-28
Source-checked
41MiMo-V2.6-FlashXiaomi2026-09-2251.3%
2026-09-28
Source-checked
42GLM-5.2Zhipu2026-06-1351.2%
2026-09-28
Source-checked
43DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek2026-08-1351%
2026-09-28
Source-checked
44MiMo-V2.5-ProXiaomi2026-04-2250.6%
2026-09-28
Source-checked
45Qwen3.8-Flash-NextAlibaba2026-08-2650.6%
2026-09-28
Source-checked
46DeepSeek V4 Flash 0731 (Reasoning, Max Effort)DeepSeek2026-07-3150.3%
2026-09-28
Source-checked
47Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)Anthropic2026-02-1750.1%
2026-09-28
Source-checked
48MiniMax M2.7MiniMax2026-04-1550.1%
2026-09-28
Source-checked
49DeepSeek V4 Flash Vision (Reasoning, Max Effort)DeepSeek2026-08-2149.7%
2026-09-28
Source-checked
50Inkling SmallOther2026-07-3049.7%
2026-09-28
Source-checked
51Qwen3.7 MaxAlibaba2026-05-1949.5%
2026-09-28
Source-checked
52Claude Opus 5 (Adaptive Reasoning, Low Effort)Anthropic2026-07-2449.2%
2026-09-28
Source-checked
53Hy3Other2026-07-0648.6%
2026-09-28
Source-checked
54Grok 4.3xAI2026-04-3048.3%
2026-09-28
Source-checked
55Solar Open2 250BOther2026-08-1248%
2026-09-28
Source-checked
56Kimi K2.7 CodeMoonshot2026-06-1247.8%
2026-09-28
Source-checked
57GPT-5.4 NanoOpenAI2026-03-1747.2%
2026-09-28
Source-checked
58MiniMax M3MiniMax2026-05-3147.1%
2026-09-28
Source-checked
59Inkling (xhigh)Other2026-07-1547%
2026-09-28
Source-checked
60Qwen3.8 27BAlibaba2026-08-1446.6%
2026-09-28
Source-checked
61Qwen 3.7 PlusAlibaba2026-06-0246.1%
2026-09-28
Source-checked
62Claude 4.5 Sonnet (Reasoning)Anthropic2025-09-2945.7%
2026-09-28
Source-checked
63Apodex 1.1Other2026-08-3045.5%
2026-09-28
Source-checked
64Gemma 4 31B ITGoogle2026-03-0145.5%
2026-09-28
Source-checked
65Ring-2.6-1TOther2026-05-0845%
2026-09-28
Source-checked
66Muse GlimmerMeta2026-08-1044.9%
2026-09-28
Source-checked
67GLM-5.1Zhipu2026-04-0744.8%
2026-09-28
Source-checked
68Qwen3.5 397B A17B (Reasoning)Alibaba2026-02-1644.8%
2026-09-28
Source-checked
69Solar Pro 4Other2026-08-0644.6%
2026-09-28
Source-checked
70MiMo-V2.5Xiaomi2026-04-2243.9%
2026-09-28
Source-checked
71Step 3.7 FlashStepFun2026-06-0143.9%
2026-09-28
Source-checked
72Gemini 3.1 Flash LiteGoogle2026-05-0743.4%
2026-09-28
Source-checked
73Nex-N2-ProOther2026-06-0243.3%
2026-09-28
Source-checked
74K2 Horizon 375B A23BOther2026-09-0342.9%
2026-09-28
Source-checked
75Qwen3.6 27B (Reasoning)Alibaba2026-04-2242.8%
2026-09-28
Source-checked
76Motif 3Other2026-08-1242.2%
2026-09-28
Source-checked
77K-EXAONE 2.0 0803Other2026-08-1242%
2026-09-28
Source-checked
78Ling 3.0 FlashOther2026-08-0442%
2026-09-28
Source-checked
79Gemini 3.5 Flash-LiteGoogle2026-07-2141.3%
2026-09-28
Source-checked
80A.X-K2Other2026-08-1241%
2026-09-28
Source-checked
81Nemotron 3 Ultra 550B A55B (Reasoning)NVIDIA2026-06-0440.3%
2026-09-28
Source-checked
82Mistral Medium 3.5Mistral2026-04-2940.2%
2026-09-28
Source-checked
83Qwen3.5 122B A10B (Reasoning)Alibaba2026-02-2439.7%
2026-09-28
Source-checked
84Grok 4.3 (Non-reasoning)xAI2026-04-3039.4%
2026-09-28
Source-checked
85North Mini CodeCohere2026-06-0938.8%
2026-09-28
Source-checked
86Granite 4.2 30BOther2026-08-2537.8%
2026-09-28
Source-checked
87G9v3-39A5BOther2026-08-0336.8%
2026-09-28
Source-checked
88Mistral Large 3Mistral2025-12-0236.6%
2026-09-28
Source-checked
89Qwen3.6 35B A3B (Reasoning)Alibaba2026-04-1636.6%
2026-09-28
Source-checked
90LongCat-2.0Meituan2026-06-2936.3%
2026-09-28
Source-checked
91Nemotron 3.5 LightningNVIDIA2026-08-1132.1%
2026-09-28
Source-checked
92Llama 4 MaverickMeta2026-04-0531.7%
2026-09-28
Source-checked
93Granite 4.2 8BOther2026-08-2531.5%
2026-09-28
Source-checked
94MiniCPM5-2BOpenBMB2026-09-0726.3%
2026-09-28
Source-checked
95Granite 4.2 3BOther2026-08-2525.3%
2026-09-28
Source-checked
96Ling 3.0 TinyOther2026-08-0624.2%
2026-09-28
Source-checked
97Llama 4 ScoutMeta2026-04-0521.3%
2026-09-28
Source-checked
Method

What it covers

This benchmark sits in the coding family. It uses Scientific coding problems with library dependencies. The score should travel with its task format, scoring method, source date, and benchmark version.

Score ceiling

Where it breaks down

No ceiling note is recorded yet. Treat clustering near the top as a warning that the benchmark may no longer separate frontier models.

Receipts

Sources and further reading

← All benchmarksHow we score

Questions about this benchmark

Every answer below is assembled from the dated fields on this page. Nothing is written separately for search.

What does SciCode measure?

Scientific programming tasks requiring domain libraries and multi-step numerical code — AA snapshot runs.

What does a high SciCode score not prove?

A strong SciCode result says nothing about:

  • Wet-lab validation — generated code is graded against reference tests only.
  • Library versions in your environment — harness pins dependency sets.

How is SciCode scored?

Percentage solved from AA runs. Task format: Scientific coding problems with library dependencies.

Is SciCode saturated?

SciCode is currently marked Active in the atlas.

Can SciCode results be contaminated by training data?

Contamination risk for SciCode is graded Medium contamination. Treat every row on this page as a public claim with a source and a date, not as a controlled experiment.

The Pack · Editorial newsletter

Know when the rules move.

One short email when a university changes its AI rule for theses, or a research tool changes its pricing or privacy terms. No tracking, no third-party analytics. Unsubscribe in one click.

The thesis AI checklist is free to use, no signup needed →