paper-with-me

Code Generation 벤치마크

Code Generation on SciCode

226개 결과 · ⬇ CSV · JSON

Accuracy

6.9 21.9 36.9 51.9 66.9 2024-12 2026-11 DeepSeek V3 (Dec '24) (DeepSeek) — 35.8 (2024-12-26) DeepSeek R1 (Jan '25) (DeepSeek) — 38.3 (2025-01-20) o3-mini (High) (OpenAI) — 42.8 (2025-01-31) Gemma 3 27B Instruct (Google) — 23.3 (2025-03-12) Gemma 3 12B Instruct (Google) — 16.4 (2025-03-12) Mistral Small 3.1 (Mistral) — 27.8 (2025-03-17) DeepSeek V3 0324 (DeepSeek) — 39.0 (2025-03-25) Llama 4 Maverick (Meta) — 31.7 (2025-04-05) Llama 4 Scout (Meta) — 21.3 (2025-04-05) Qwen3 32B (Reasoning) (Alibaba) — 36.0 (2025-04-28) Qwen3 14B (Reasoning) (Alibaba) — 30.7 (2025-04-28) Qwen3 8B (Reasoning) (Alibaba) — 27.9 (2025-04-28) Gemini 2.5 Pro (Google) — 46.3 (2025-06-05) Mistral Small 3.2 (Mistral) — 28.6 (2025-06-20) Qwen3 235B A22B 2507 (Reasoning) (Alibaba) — 41.4 (2025-07-25) Qwen3 30B A3B 2507 (Reasoning) (Alibaba) — 33.0 (2025-07-30) gpt-oss-20b (High) (OpenAI) — 38.9 (2025-08-05) gpt-oss-120b (High) (OpenAI) — 34.0 (2025-08-05) GPT-5 mini (High) (OpenAI) — 39.0 (2025-08-07) Mistral Medium 3.1 (Mistral) — 32.4 (2025-08-12) DeepSeek V3.1 Terminus (Reasoning) (DeepSeek) — 38.0 (2025-09-22) Claude 4.5 Sonnet (Reasoning) (Anthropic) — 45.7 (2025-09-29) Claude 4.5 Haiku (Reasoning) (Anthropic) — 42.2 (2025-10-15) Mistral Large 3 (Mistral) — 36.6 (2025-12-02) Ministral 3 14B (Mistral) — 23.8 (2025-12-02) Ministral 3 8B (Mistral) — 20.7 (2025-12-02) Ministral 3 3B (Mistral) — 15.3 (2025-12-02) Devstral 2 (Mistral) — 32.8 (2025-12-09) Devstral Small 2 (Mistral) — 32.4 (2025-12-09) NVIDIA Nemotron 3 Nano 30B A3B (Reasoning) (NVIDIA) — 30.6 (2025-12-15) Qwen3 Coder Next (Alibaba) — 36.2 (2026-02-03) Qwen3.5 397B A17B (Reasoning) (Alibaba) — 44.8 (2026-02-16) Claude Sonnet 4.6 (Max) (Anthropic) — 50.1 (2026-02-17) Gemini 3.1 Pro Preview (Google) — 58.7 (2026-02-19) Mercury 2 (Inception) — 37.7 (2026-02-20) Qwen3.5 122B A10B (Reasoning) (Alibaba) — 39.7 (2026-02-24) Qwen3.5 9B (Reasoning) (Alibaba) — 29.5 (2026-03-02) Gemini 3.1 Flash-Lite (Google) — 43.4 (2026-03-03) Nemotron 3 Super 120B A12B (Reasoning) (NVIDIA) — 36.2 (2026-03-11) Mistral Small 4 (Reasoning) (Mistral) — 38.8 (2026-03-16) NVIDIA Nemotron 3 Nano 4B (NVIDIA) — 16.1 (2026-03-16) GPT-5.4 mini (Xhigh) (OpenAI) — 52.1 (2026-03-17) GPT-5.4 nano (Xhigh) (OpenAI) — 47.2 (2026-03-17) MiniMax-M2.7 (MiniMax) — 50.1 (2026-03-18) Trinity Large Thinking (Arcee AI) — 40.6 (2026-04-01) Gemma 4 31B (Reasoning) (Google) — 45.5 (2026-04-02) Gemma 4 26B A4B (Reasoning) (Google) — 40.0 (2026-04-02) Gemma 4 E4B (Reasoning) (Google) — 24.4 (2026-04-03) Solar Pro 3 (Upstage) — 25.5 (2026-04-06) GLM-5.1 (Reasoning) (Z AI) — 44.8 (2026-04-07) Qwen3.6 35B A3B (Reasoning) (Alibaba) — 36.6 (2026-04-16) Kimi K2.6 (Reasoning) (Kimi) — 51.5 (2026-04-20) MiMo-V2.5-Pro (Reasoning) (Xiaomi) — 50.6 (2026-04-22) MiMo-V2.5 (Xiaomi) — 43.9 (2026-04-22) Qwen3.6 27B (Reasoning) (Alibaba) — 42.8 (2026-04-22) GPT-5.5 (High) (OpenAI) — 56.1 (2026-04-23) GPT-5.5 (Xhigh) (OpenAI) — 55.8 (2026-04-23) GPT-5.5 (Medium) (OpenAI) — 54.5 (2026-04-23) DeepSeek V4 Pro 0424 (Max) (DeepSeek) — 50.8 (2026-04-24) DeepSeek V4 Flash 0420 (Max) (DeepSeek) — 45.3 (2026-04-24) DeepSeek V4 Flash 0420 (High) (DeepSeek) — 40.2 (2026-04-24) Mistral Medium 3.5 (Mistral) — 40.2 (2026-04-29) Grok 4.3 (High) (SpaceXAI) — 48.3 (2026-04-30) Grok 4.3 (Non-reasoning) (SpaceXAI) — 39.4 (2026-04-30) Ring-2.6-1T (InclusionAI) — 45.0 (2026-05-08) Gemini 3.5 Flash (High) (Google) — 53.9 (2026-05-19) Qwen3.7 Max (Alibaba) — 49.5 (2026-05-19) Command A+ (Cohere) — 38.5 (2026-05-20) Claude Opus 4.8 (Max) (Anthropic) — 54.4 (2026-05-28) Step 3.7 Flash (StepFun) — 43.9 (2026-05-29) MiniMax-M3 (MiniMax) — 47.1 (2026-06-01) Qwen3.7 Plus (Alibaba) — 46.1 (2026-06-01) Nex-N2-Pro (Based on Qwen3.5-397B-A17B) (Nex AGI) — 43.3 (2026-06-02) Nemotron 3 Ultra 550B A55B (Reasoning) (NVIDIA) — 40.3 (2026-06-04) Claude Fable 5 (Max, Opus 4.8 Fallback) (Anthropic) — 61.0 (2026-06-09) North Mini Code (Cohere) — 38.8 (2026-06-09) Kimi K2.7 Code (Kimi) — 47.8 (2026-06-12) GLM-5.2 (Max) (Z AI) — 51.2 (2026-06-16) GPT-5.5 Instant (June 2026) (OpenAI) — 52.5 (2026-06-25) LongCat 2.0 (LongCat) — 36.3 (2026-06-29) Claude Sonnet 5 (Max) (Anthropic) — 54.3 (2026-06-30) Claude Sonnet 5 (High) (Anthropic) — 54.3 (2026-06-30) Claude Sonnet 5 (Xhigh) (Anthropic) — 54.1 (2026-06-30) Claude Sonnet 5 (Medium) (Anthropic) — 51.6 (2026-06-30) Claude Sonnet 5 (Low) (Anthropic) — 50.1 (2026-06-30) Hy3 (Tencent) — 48.6 (2026-07-06) Grok 4.5 (High) (SpaceXAI) — 55.0 (2026-07-08) Muse Spark 1.1 (Xhigh) (Meta) — 58.8 (2026-07-09) GPT-5.6 Sol (High) (OpenAI) — 57.8 (2026-07-09) GPT-5.6 Sol (Medium) (OpenAI) — 57.4 (2026-07-09) GPT-5.6 Sol (Max) (OpenAI) — 57.1 (2026-07-09) GPT-5.6 Sol (Xhigh) (OpenAI) — 57.1 (2026-07-09) GPT-5.6 Sol (Low) (OpenAI) — 56.4 (2026-07-09) GPT-5.6 Terra (Max) (OpenAI) — 55.0 (2026-07-09) GPT-5.6 Luna (Max) (OpenAI) — 53.6 (2026-07-09) GPT-5.6 Terra (High) (OpenAI) — 52.4 (2026-07-09) GPT-5.6 Terra (Xhigh) (OpenAI) — 52.3 (2026-07-09) JT-4.1 Flash 236B A21B (Reasoning) (China Mobile) — 52.0 (2026-07-09) GPT-5.6 Luna (High) (OpenAI) — 51.6 (2026-07-09) GPT-5.6 Luna (Xhigh) (OpenAI) — 50.5 (2026-07-09) GPT-5.6 Terra (Medium) (OpenAI) — 50.5 (2026-07-09) GPT-5.6 Terra (Low) (OpenAI) — 49.9 (2026-07-09) GPT-5.6 Sol (Non-reasoning) (OpenAI) — 47.7 (2026-07-09) GPT-5.6 Luna (Medium) (OpenAI) — 46.8 (2026-07-09) GPT-5.6 Luna (Low) (OpenAI) — 46.1 (2026-07-09) GPT-5.6 Terra (Non-reasoning) (OpenAI) — 45.1 (2026-07-09) GPT-5.6 Luna (Non-reasoning) (OpenAI) — 40.4 (2026-07-09) Inkling (Xhigh) (Thinking Machines) — 47.0 (2026-07-15) Kimi K3 (Max) (Kimi) — 59.5 (2026-07-16) Kimi K3 (Low) (Kimi) — 52.7 (2026-07-16) Gemini 3.6 Flash (High) (Google) — 53.4 (2026-07-21) Gemini 3.5 Flash-Lite (Google) — 41.3 (2026-07-21) Claude Opus 5 (Max) (Anthropic) — 56.4 (2026-07-24) Claude Opus 5 (Xhigh) (Anthropic) — 55.7 (2026-07-24) Claude Opus 5 (High) (Anthropic) — 55.4 (2026-07-24) Claude Opus 5 (Medium) (Anthropic) — 51.5 (2026-07-24) Claude Opus 5 (Low) (Anthropic) — 49.2 (2026-07-24) Celeris-1 (Celeris) — 21.6 (2026-07-24) Inkling Small (Thinking Machines) — 49.7 (2026-07-30) DeepSeek V4 Flash 0731 (Max) (DeepSeek) — 50.3 (2026-07-31) Qwen3.8 Max (Alibaba) — 53.2 (2026-08-03) Ling 3.0 Flash (InclusionAI) — 42.0 (2026-08-04) LFM2.5-2.6B (Liquid AI) — 14.4 (2026-08-04) Muse Spark 1.2 (Xhigh) (Meta) — 57.4 (2026-08-05) Solar Pro 4 (Upstage) — 44.6 (2026-08-06) Ling 3.0 Tiny (InclusionAI) — 24.2 (2026-08-06) Quasar 438B (Max, Based on GLM-5.2) (Multiverse Computing) — 48.1 (2026-08-10) Muse Glimmer (High) (Meta) — 44.9 (2026-08-10) Nemotron 3.5 Lightning (NVIDIA) — 32.1 (2026-08-11) Grok 4.6 (High) (SpaceXAI) — 56.5 (2026-08-12) Grok 4.6 (Medium) (SpaceXAI) — 55.9 (2026-08-12) Qwen3.8 2.4T A95B (Alibaba) — 54.1 (2026-08-12) Grok 4.6 (Xhigh) (SpaceXAI) — 53.0 (2026-08-12) Grok 4.6 (Low) (SpaceXAI) — 49.4 (2026-08-12) Solar Open2 250B (Upstage) — 48.0 (2026-08-12) Motif 3 (Motif Technologies) — 42.2 (2026-08-12) K-EXAONE 2.0 0803 (LG AI Research) — 42.0 (2026-08-12) A.X-K2 (SK Telecom) — 41.0 (2026-08-12) Gemini 3.7 Flash (Medium) (Google) — 59.8 (2026-08-13) Gemini 3.7 Flash (High) (Google) — 57.2 (2026-08-13) Gemini 3.7 Flash (Low) (Google) — 55.7 (2026-08-13) DeepSeek V4 Pro 0813 (Max) (DeepSeek) — 51.0 (2026-08-13) DeepSeek V4 Pro 0813 (Non-reasoning) (DeepSeek) — 39.9 (2026-08-13) Qwen3.8 27B (Xhigh) (Alibaba) — 46.6 (2026-08-14) Qwen3.8 27B (Low) (Alibaba) — 40.0 (2026-08-14) Qwen3.8 27B (Medium) (Alibaba) — 39.0 (2026-08-14) Qwen3.8 27B (Non-reasoning) (Alibaba) — 36.2 (2026-08-14) GLM-5.3 (Max) (Z AI) — 59.0 (2026-08-18) GLM-5.3 (Low) (Z AI) — 42.0 (2026-08-18) G9v3-39A5B (AI9Stars) — 36.8 (2026-08-20) DeepSeek V4 Flash Vision (Max) (DeepSeek) — 49.7 (2026-08-21) Granite 4.2 30B (IBM) — 37.8 (2026-08-25) Granite 4.2 8B (IBM) — 31.5 (2026-08-25) Granite 4.2 3B (IBM) — 25.3 (2026-08-25) GLM 5.3 Flash (Z AI) — 51.6 (2026-08-26) Qwen3.8-Flash-Next (Alibaba) — 50.6 (2026-08-26) Apodex 1.1 (Apodex) — 45.5 (2026-08-30) Claude Fable 5.1 (Max, Default Fallback) (Anthropic) — 63.1 (2026-09-01) Claude Fable 5.1 (Xhigh, Default Fallback) (Anthropic) — 60.9 (2026-09-01) Claude Fable 5.1 (High, Default Fallback) (Anthropic) — 58.7 (2026-09-01) Claude Fable 5.1 (Low, Default Fallback) (Anthropic) — 56.7 (2026-09-01) Claude Fable 5.1 (Medium, Default Fallback) (Anthropic) — 56.4 (2026-09-01) Muse Spark 1.3 (Xhigh) (Meta) — 59.7 (2026-09-02) Muse Spark 1.3 (Max) (Meta) — 58.8 (2026-09-02) Gemini 3.8 Flash (High) (Google) — 56.6 (2026-09-02) Gemini 3.8 Flash (Medium) (Google) — 55.1 (2026-09-02) Gemini 3.8 Flash (Low) (Google) — 55.0 (2026-09-02) Qwen3.8 Max (0902) (Alibaba) — 52.1 (2026-09-02) GPT-6 Astra (Max) (OpenAI) — 56.5 (2026-09-03) GPT-6 Astra (Xhigh) (OpenAI) — 55.7 (2026-09-03) GPT-6 Astra (High) (OpenAI) — 55.4 (2026-09-03) GPT-6 Astra (Medium) (OpenAI) — 54.2 (2026-09-03) GPT-6 Astra (Low) (OpenAI) — 54.1 (2026-09-03) K2 Horizon 375B A23B (Institute of Foundation Models) — 42.9 (2026-09-03) K2 Horizon MoVA 36B A4B (Institute of Foundation Models) — 40.0 (2026-09-03) K2 Horizon 7B (Institute of Foundation Models) — 27.5 (2026-09-03) K2 Horizon 3.7B (Institute of Foundation Models) — 22.0 (2026-09-03) K2 Horizon 0.9B (Institute of Foundation Models) — 6.9 (2026-09-03) MiniCPM5-2B (OpenBMB) — 26.3 (2026-09-07) Mercury 2.5 (Inception) — 38.5 (2026-09-08) DeepSeek V4.1 Flash (Max) (DeepSeek) — 51.9 (2026-09-10) Ling-3.0-flash-VL (InclusionAI) — 44.2 (2026-09-10) DeepSeek V4.1 Flash (Non-reasoning) (DeepSeek) — 35.5 (2026-09-10) Ling-3.0-flash-Fin (InclusionAI) — 42.4 (2026-09-11) Step 5 Preview (StepFun) — 58.9 (2026-09-18) MiMo-V2.6-Pro (Xiaomi) — 60.9 (2026-09-21) Grok 4.7 (High) (SpaceXAI) — 57.8 (2026-09-21) Grok 4.7 (Xhigh) (SpaceXAI) — 57.4 (2026-09-21) Grok 4.7 (Low) (SpaceXAI) — 54.9 (2026-09-21) MiMo-V2.6-Flash (Xiaomi) — 51.3 (2026-09-21) Claude Opus 5.5 (Max, Default Fallback) (Anthropic) — 66.9 (2026-09-22) Claude Opus 5.5 (Xhigh, Default Fallback) (Anthropic) — 65.0 (2026-09-22) Claude Opus 5.5 (High, Default Fallback) (Anthropic) — 60.4 (2026-09-22) Claude Opus 5.5 (Medium, Default Fallback) (Anthropic) — 59.3 (2026-09-22) Claude Opus 5.5 (Low, Default Fallback) (Anthropic) — 58.6 (2026-09-22) GPT-6 Sol (Max) (OpenAI) — 57.6 (2026-09-22) GPT-6 Sol (Xhigh) (OpenAI) — 55.1 (2026-09-22) GPT-6 Sol (High) (OpenAI) — 54.9 (2026-09-22) GPT-6 Luna (Max) (OpenAI) — 54.6 (2026-09-22) GPT-6 Sol (Medium) (OpenAI) — 53.8 (2026-09-22) GPT-6 Luna (Xhigh) (OpenAI) — 51.7 (2026-09-22) GPT-6 Luna (Medium) (OpenAI) — 50.9 (2026-09-22) GPT-6 Luna (High) (OpenAI) — 50.3 (2026-09-22) GPT-6 Sol (Low) (OpenAI) — 50.2 (2026-09-22) Solar Mini 4 (Upstage) — 47.6 (2026-09-22) GPT-6 Sol (Non-reasoning) (OpenAI) — 47.3 (2026-09-22) GPT-6 Luna (Low) (OpenAI) — 46.9 (2026-09-22) GPT-6 Luna (Non-reasoning) (OpenAI) — 43.1 (2026-09-22) Claude Sonnet 5.5 (Max, Default Fallback) (Anthropic) — 61.0 (2026-09-28) Claude Sonnet 5.5 (Xhigh, Default Fallback) (Anthropic) — 57.3 (2026-09-28) Claude Sonnet 5.5 (High, Default Fallback) (Anthropic) — 53.7 (2026-09-28) Claude Sonnet 5.5 (Medium, Default Fallback) (Anthropic) — 52.9 (2026-09-28) Claude Sonnet 5.5 (Low, Default Fallback) (Anthropic) — 49.1 (2026-09-28) GPT-6.1 Sol (High) (OpenAI) — 55.8 (2026-09-29) GPT-6.1 Sol (Xhigh) (OpenAI) — 55.7 (2026-09-29) GPT-6.1 Sol (Max) (OpenAI) — 54.2 (2026-09-29) GPT-6.1 Sol (Medium) (OpenAI) — 53.2 (2026-09-29) GPT-6.1 Sol (Low) (OpenAI) — 53.2 (2026-09-29) Gemini 4 Argon (High) (Google) — 61.8 (2026-09-30) Ling 3.1 Flash (InclusionAI) — 54.1 (2026-10-01) Mistral Large 4 Preview (Mistral) — 54.2 (2026-10-06) Claude Haiku 5.5 (Max) (Anthropic) — 55.0 (2026-10-07) Claude Haiku 5.5 (Xhigh) (Anthropic) — 51.7 (2026-10-07) Claude Haiku 5.5 (Low) (Anthropic) — 49.2 (2026-10-07) Claude Haiku 5.5 (Medium) (Anthropic) — 49.0 (2026-10-07) Claude Haiku 5.5 (High) (Anthropic) — 48.7 (2026-10-07) DeepSeek V3 (Dec '24) (DeepSeek) — 35.8 (2024-12-26) DeepSeek R1 (Jan '25) (DeepSeek) — 38.3 (2025-01-20) o3-mini (High) (OpenAI) — 42.8 (2025-01-31) Gemini 2.5 Pro (Google) — 46.3 (2025-06-05) Claude Sonnet 4.6 (Max) (Anthropic) — 50.1 (2026-02-17) Gemini 3.1 Pro Preview (Google) — 58.7 (2026-02-19) Claude Fable 5 (Max, Opus 4.8 Fallback) (Anthropic) — 61.0 (2026-06-09) Claude Fable 5.1 (Max, Default Fallback) (Anthropic) — 63.1 (2026-09-01) Claude Opus 5.5 (Max, Default Fallback) (Anthropic) — 66.9 (2026-09-22)
RankModel Accuracy PaperCodeYear
1 Claude Opus 5.5 (Max, Default Fallback) (Anthropic) 외부 66.9 artificialanalysis.ai 2026
2 Claude Opus 5.5 (Xhigh, Default Fallback) (Anthropic) 외부 65 artificialanalysis.ai 2026
3 Claude Fable 5.1 (Max, Default Fallback) (Anthropic) 외부 63.1 artificialanalysis.ai 2026
4 Gemini 4 Argon (High) (Google) 외부 61.8 artificialanalysis.ai 2026
5 Claude Sonnet 5.5 (Max, Default Fallback) (Anthropic) 외부 61 artificialanalysis.ai 2026
5 Claude Fable 5 (Max, Opus 4.8 Fallback) (Anthropic) 외부 61 artificialanalysis.ai 2026
7 Claude Fable 5.1 (Xhigh, Default Fallback) (Anthropic) 외부 60.9 artificialanalysis.ai 2026
7 MiMo-V2.6-Pro (Xiaomi) 외부 60.9 artificialanalysis.ai 2026
9 Claude Opus 5.5 (High, Default Fallback) (Anthropic) 외부 60.4 artificialanalysis.ai 2026
10 Gemini 3.7 Flash (Medium) (Google) 외부 59.8 artificialanalysis.ai 2026
11 Muse Spark 1.3 (Xhigh) (Meta) 외부 59.7 artificialanalysis.ai 2026
12 Kimi K3 (Max) (Kimi) 외부 59.5 artificialanalysis.ai 2026
13 Claude Opus 5.5 (Medium, Default Fallback) (Anthropic) 외부 59.3 artificialanalysis.ai 2026
14 GLM-5.3 (Max) (Z AI) 외부 59 artificialanalysis.ai 2026
15 Step 5 Preview (StepFun) 외부 58.9 artificialanalysis.ai 2026
16 Muse Spark 1.3 (Max) (Meta) 외부 58.8 artificialanalysis.ai 2026
16 Muse Spark 1.1 (Xhigh) (Meta) 외부 58.8 artificialanalysis.ai 2026
18 Claude Fable 5.1 (High, Default Fallback) (Anthropic) 외부 58.7 artificialanalysis.ai 2026
18 Gemini 3.1 Pro Preview (Google) 외부 58.7 artificialanalysis.ai 2026
20 Claude Opus 5.5 (Low, Default Fallback) (Anthropic) 외부 58.6 artificialanalysis.ai 2026
1–20 / 226 다음 → 페이지당 10 20 50 100