paper-with-me

Code Generation 벤치마크

Code Generation on MBPP

101개 결과 · ⬇ CSV · JSON

Accuracy

5.5 28.27 51.05 73.82 96.6 2022-04 2026-09 PaLM Coder 540B — 47.0 (2022-04-05) PaLM 540B — 36.8 (2022-04-05) InCoder 6.7B (0-shot) — 19.4 (2022-04-12) code-davinci-001 175B + MBR-Exec — 58.2 (2022-04-25) code-davinci-002 175B + CodeT — 67.7 (2022-07-21) code-davinci-001 175B + CodeT — 61.9 (2022-07-21) code-cushman-001 12B (CodeT) — 55.4 (2022-07-21) CodeGen-Mono 16B + CodeT — 49.5 (2022-07-21) InCoder 6.7B + CodeT — 34.4 (2022-07-21) code-davinci-002 175B + Reviewer — 66.9 (2022-11-29) code-davinci-002 175B + Coder-Reviewer — 66.4 (2022-11-29) code-davinci-002 175B + MBR-Exec — 63.0 (2022-11-29) code-cushman-001 12B + MBR-Exec — 48.3 (2022-11-29) CodeGen 16B + MBR-Exec — 47.3 (2022-11-29) CodeGen 16B + Coder-Reviewer — 46.2 (2022-11-29) CodeGen 16B + Reviewer — 44.1 (2022-11-29) InCoder 6.7B + MBR-Exec — 26.7 (2022-11-29) InCoder 6.7B + Coder-Reviewer — 26.1 (2022-11-29) InCoder 6.7B + Reviewer — 24.4 (2022-11-29) code-davinci-002 175B + LEVER — 68.9 (2023-02-16) LLaMA 65B (0-shot) — 37.7 (2023-02-27) LLaMA 33B (0-shot) — 30.2 (2023-02-27) LLaMA 13B (0-shot) — 22.0 (2023-02-27) LLaMA 7B (0-shot) — 17.7 (2023-02-27) CodeGeeX-13B — 24.4 (2023-03-30) GPT-4 (Self-Debugging with unit tests + trace) — 80.2 (2023-04-11) GPT-3.5 Turbo (Self-Debugging with unit tests + trace) — 72.8 (2023-04-11) code-davinci-002 175B (Self-Debugging with unit tests + trace) — 70.8 (2023-04-11) GPT-3.5 Turbo (3-shot) — 67.6 (2023-04-11) code-davinci-002 175B (3-shot) — 61.4 (2023-04-11) StarCoder 15.5B (Self-Debugging with unit tests + trace) — 53.2 (2023-04-11) StarCoder 15.5B (3-shot) — 47.2 (2023-04-11) StarCoder 15.5B — 52.7 (2023-05-09) StarCoderBase 15.5B — 49.0 (2023-05-09) SantaCoder 1.1B — 35.0 (2023-05-09) PaLM 2-S* (few-shot) — 50.0 (2023-05-17) WizardCoder 15B — 51.8 (2023-06-14) Llama 2 70B (zero-shot) — 45.0 (2023-07-18) Llama 2 34B (0-shot) — 33.0 (2023-07-18) Llama 2 13B (0-shot) — 30.6 (2023-07-18) Llama 2 7B (0-shot) — 20.8 (2023-07-18) GPT-4 (ChatGPT Plus) — 87.5 (2023-07-24) GPT-3.5 Turbo (ChatGPT) — 83.2 (2023-07-24) GPT-4 (Bing Chat) — 82.0 (2023-07-24) Bard (PaLM 2/chat-bison-001) — 76.2 (2023-07-24) Claude — 71.4 (2023-07-24) Code Llama - Python 70B (3-shot) — 65.5 (2023-08-24) Code Llama 70B (3-shot) — 62.4 (2023-08-24) Code Llama - Instruct 70B (3-shot) — 62.2 (2023-08-24) Unnatural Code Llama 34B (3-shot) — 61.2 (2023-08-24) Code Llama - Instruct 34B (3-shot) — 57.0 (2023-08-24) Code Llama - Python 34B (3-shot) — 56.2 (2023-08-24) Code Llama 34B (3-shot) — 55.0 (2023-08-24) GPT-3.5 Turbo — 52.2 (2023-08-24) Code Llama - Instruct 13B (3-shot) — 49.4 (2023-08-24) Code Llama - Python 13B (3-shot) — 49.0 (2023-08-24) Code Llama - Python 7B (3-shot) — 47.6 (2023-08-24) Code Llama 13B (3-shot) — 47.0 (2023-08-24) Code Llama - Instruct 7B (3-shot) — 44.4 (2023-08-24) Code Llama 7B (3-shot) — 41.4 (2023-08-24) phi-1.5-web 1.3B — 43.5 (2023-09-11) o1-mini + Language Agent Tree Search (Hamming.ai) — 82.3 (2023-10-06) GPT-3.5 Turbo + Language Agent Tree Search — 81.1 (2023-10-06) Mistral 7B (3-shot) — 47.5 (2023-10-10) GPT-3.5 Turbo + INTERVENOR — 69.8 (2023-11-16) GPT-3.5 Turbo (few-shot) — 45.4 (2023-11-16) GPT-3.5 Turbo (0-shot) — 39.8 (2023-11-16) GPT-4 + AgentCoder — 91.8 (2023-12-20) GPT-3.5 Turbo (ChatGPT) + AgentCoder — 89.9 (2023-12-20) Qwen2idae-16x14B (4-shot) — 48.6 (2024-01-05) Camelidae-8×34B (4-shot) — 41.4 (2024-01-05) Mixtral 8x7B (3-shot) — 60.7 (2024-01-08) GPT-4 (few-shot) — 80.0 (2024-01-25) GPT-3.5 Turbo (few-shot) — 70.8 (2024-01-25) DeepSeek-Coder-Instruct 33B (few-shot) — 70.0 (2024-01-25) DeepSeek-Coder-Base 33B (few-shot) — 66.0 (2024-01-25) DeepSeek-Coder-Instruct 6.7B (few-shot) — 65.4 (2024-01-25) DeepSeek-Coder-Base 6.7B (few-shot) — 60.6 (2024-01-25) DeepSeek-Coder-Instruct 1.3B (few-shot) — 49.4 (2024-01-25) DeepSeek-Coder-Base 1.3B (few-shot) — 46.2 (2024-01-25) StarCoder2-15B — 66.2 (2024-02-29) Claude 3 Opus — 86.4 (2024-03-04) Claude 3 Haiku — 80.4 (2024-03-04) Claude 3 Sonnet — 79.4 (2024-03-04) Branch-Train-Merge 4x7B (top-2) — 42.6 (2024-03-12) Branch-Train-MiX 4x7B (sampling top-2 experts) — 39.4 (2024-03-12) GPT-3.5 Turbo + FlowGenScrum + Test — 83.8 (2024-03-23) o1-mini + MapCoder (Hamming.ai) — 93.2 (2024-05-18) MapCoder (GPT-4o) — 89.7 (2024-05-18) MapCoder (GPT-4) — 83.1 (2024-05-18) MGDebugger (DeepSeek-V3-0324) — 92.4 (2024-10-02) MGDebugger (CodeQwen1.5) — 80.8 (2024-10-02) AFlow(GPT-4o-mini) — 83.4 (2024-10-14) LPW (GPT-4o) — 84.8 (2024-11-21) QualityFlow (Sonnet-3.5) — 94.2 (2025-01-20) CodeSim (GPT4o) — 90.7 (2025-02-08) EG-CFG (DeepSeek-V3-0324) — 96.6 (2025-06-12) EG-CFG (DeepSeek Coder 1.3b Instruct) — 83.2 (2025-06-12) Programming — 34.0 (2026-01-28) Group-Based — 5.5 (2026-02-04) PaLM Coder 540B — 47.0 (2022-04-05) code-davinci-001 175B + MBR-Exec — 58.2 (2022-04-25) code-davinci-002 175B + CodeT — 67.7 (2022-07-21) code-davinci-002 175B + LEVER — 68.9 (2023-02-16) GPT-4 (Self-Debugging with unit tests + trace) — 80.2 (2023-04-11) GPT-4 (ChatGPT Plus) — 87.5 (2023-07-24) GPT-4 + AgentCoder — 91.8 (2023-12-20) o1-mini + MapCoder (Hamming.ai) — 93.2 (2024-05-18) QualityFlow (Sonnet-3.5) — 94.2 (2025-01-20) EG-CFG (DeepSeek-V3-0324) — 96.6 (2025-06-12)
RankModel Accuracy Extra Training Data PaperCodeYear
1 EG-CFG (DeepSeek-V3-0324) 96.6 Execution Guided Line-by-Line Code Generation boazlavon/eg_cfg 2025
2 QualityFlow (Sonnet-3.5) 94.2 QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks 2025
3 o1-mini + MapCoder (Hamming.ai) 93.2 ✓ MapCoder: Multi-Agent Code Generation for Competitive Problem Solving md-ashraful-pramanik/mapcoder · Luoji-zju/Agents4PLC_release 2024
4 MGDebugger (DeepSeek-V3-0324) 92.4 From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging YerbaPage/MGDebugger 2024
5 GPT-4 + AgentCoder 91.8 AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation huangd1999/AgentCoder 2023
6 CodeSim (GPT4o) 90.7 CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging kagnlp/CodeGenerator 2025
7 Jiutian-大模型 90.0
8 GPT-3.5 Turbo (ChatGPT) + AgentCoder 89.9 AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation huangd1999/AgentCoder 2023
9 MapCoder (GPT-4o) 89.7 MapCoder: Multi-Agent Code Generation for Competitive Problem Solving md-ashraful-pramanik/mapcoder · Luoji-zju/Agents4PLC_release 2024
10 GPT-4 (ChatGPT Plus) 87.5 How Does Naming Affect LLMs on Code Analysis Tasks? 2023
11 Claude 3 Opus 86.4 The Claude 3 Model Family: Opus, Sonnet, Haiku 2024
12 LPW (GPT-4o) 84.8 Planning-Driven Programming: A Large Language Model Programming Workflow you68681/lpw 2024
13 GPT-3.5 Turbo + FlowGenScrum + Test 83.8±0.6 SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents 2024
14 AFlow(GPT-4o-mini) 83.4 AFlow: Automating Agentic Workflow Generation geekan/metagpt · evoagentx/evoagentx · qixucen/atom · +1 2024
15 GPT-3.5 Turbo (ChatGPT) 83.2 How Does Naming Affect LLMs on Code Analysis Tasks? 2023
15 EG-CFG (DeepSeek Coder 1.3b Instruct) 83.2 Execution Guided Line-by-Line Code Generation boazlavon/eg_cfg 2025
17 MapCoder (GPT-4) 83.1 MapCoder: Multi-Agent Code Generation for Competitive Problem Solving md-ashraful-pramanik/mapcoder · Luoji-zju/Agents4PLC_release 2024
18 o1-mini + Language Agent Tree Search (Hamming.ai) 82.3 Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models lapisrocks/languageagenttreesearch · andyz245/LanguageAgentTreeSearch 2023
19 GPT-4 (Bing Chat) 82 How Does Naming Affect LLMs on Code Analysis Tasks? 2023
20 GPT-3.5 Turbo + Language Agent Tree Search 81.1 Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models lapisrocks/languageagenttreesearch · andyz245/LanguageAgentTreeSearch 2023
1–20 / 101 다음 → 페이지당 10 20 50 100