paper-with-me

Arithmetic Reasoning 벤치마크

Arithmetic Reasoning on GSM8K

166개 결과 · ⬇ CSV · JSON

Accuracy

4.1 27.51 50.91 74.31 97.72 2022-03 2026-09 PaLM 540B maj1@40 (8-shot) — 74.4 (2022-03-21) UL2 20B (chain-of-thought) — 4.4 (2022-05-10) UL2 20B (0-shot) — 4.1 (2022-05-10) code-davinci-002 (Least-to-Most Prompting) — 68.01 (2022-05-21) PaLM-540B (few-Shot-cot) — 58.1 (2022-05-24) Finetuned GPT-3 175B + verifier — 55.0 (2022-05-24) Text-davinci-002-175B (zero-plus-few-Shot-cot (8 samples)) — 51.5 (2022-05-24) text-davinci-002 175B (2-shot, CoT) — 41.3 (2022-05-24) text-davinci-002 175B (0-shot, CoT) — 40.7 (2022-05-24) PaLM 540B (few-shot) — 17.9 (2022-05-24) Text-davinci-002-175B (0-shot) — 10.4 (2022-05-24) GPT-Neo-2.7B + Self-Sampling — 19.5 (2022-05-28) GPT-Neo 125M + Self-Sampling — 7.5 (2022-05-28) DIVERSE 175B (8-shot) — 83.2 (2022-06-06) Minerva 62B (maj5@100) — 89.0 (2022-06-29) Minerva 540B (CoT) — 78.5 (2022-06-29) Minerva 62B (maj1@100) — 68.5 (2022-06-29) Minerva 8B (maj5@100) — 56.8 (2022-06-29) PaLM 540B (8-shot) — 56.5 (2022-06-29) Minerva 62B (8-shot) — 52.4 (2022-06-29) PaLM 62B (8-shot) — 33.0 (2022-06-29) Minerva 8B-maj1@k (8-shot) — 28.4 (2022-06-29) Minerva 8B (8-shot) — 16.2 (2022-06-29) PaLM 8B (8-shot) — 4.1 (2022-06-29) PaLM 540B (Self Improvement, Self Consistency) — 82.1 (2022-10-20) PaLM 540B (Self Consistency) — 74.4 (2022-10-20) PaLM 540B (Self Improvement, CoT Prompting) — 73.5 (2022-10-20) U-PaLM — 58.5 (2022-10-20) PaLM 540B (CoT Prompting) — 56.5 (2022-10-20) PaLM 540B (Self Improvement, Standard-Prompting) — 32.2 (2022-10-20) GPT-2-Medium 355M + question-solution classifier (BS=5) — 20.8 (2022-10-20) GPT-2-Medium 355M (fine-tuned, BS=5) — 18.3 (2022-10-20) PaLM 540B (Standard-Prompting) — 17.9 (2022-10-20) GPT-2-Medium 355M + question-solution classifier (BS=1) — 16.8 (2022-10-20) GPT-2-Medium 355M (BS=5) — 12.2 (2022-10-20) GPT-J (CoRe) — 63.2 (2022-10-28) DeepMind 70B Model (SFT+ORM-RL, ORM reranking) — 87.3 (2022-11-25) DeepMind 70B Model (SFT+PRM-RL, PRM reranking) — 87.1 (2022-11-25) DeepMind 70B Model (STaR, maj1@96) — 76.5 (2022-11-25) code-davinci-002 175B (LEVER, 8-shot) — 84.5 (2023-02-16) LLaMA 65B-maj1@k — 69.7 (2023-02-27) LLaMA 33B-maj1@k — 53.1 (2023-02-27) LLaMA 65B — 50.9 (2023-02-27) LLaMA 33B — 35.6 (2023-02-27) LLaMA 13B-maj1@k — 29.3 (2023-02-27) LLaMA 7B (maj1@k) — 18.1 (2023-02-27) LLaMA 13B — 17.8 (2023-02-27) LLaMA 7B — 11.0 (2023-02-27) GPT-3.5 (few-shot, k=5) — 57.1 (2023-03-15) GPT-4 — 87.1 (2023-03-22) CodeT5+ — 73.8 (2023-05-13) PaLM 2 (few-shot, k=8, SC) — 91.0 (2023-05-17) PaLM 2 (few-shot, k=8, CoT) — 80.7 (2023-05-17) Claude 2 (0-shot chain-of-thought) — 88.0 (2023-07-11) Claude 1.3 (0-shot chain-of-thought) — 85.2 (2023-07-11) Claude Instant 1.1 (0-shot chain-of-thought) — 80.9 (2023-07-11) LLaMA 2 70B (on-shot) — 56.8 (2023-07-18) RFT 70B — 64.8 (2023-08-03) RFT 13B — 55.3 (2023-08-03) RFT 7B — 51.2 (2023-08-03) WizardMath-7B-V1.1 — 83.2 (2023-08-18) WizardMath-70B-V1.0 — 81.6 (2023-08-18) WizardMath-13B-V1.0 — 63.9 (2023-08-18) WizardMath-7B-V1.0 — 54.9 (2023-08-18) OpenChat-3.5 7B — 77.3 (2023-09-20) MetaMath 70B — 82.3 (2023-09-21) MetaMath-Mistral-7B — 77.7 (2023-09-21) MetaMath 13B — 71.0 (2023-09-21) MetaMath 7B — 66.4 (2023-09-21) ToRA-70B (SC, k=50) — 88.3 (2023-09-29) ToRA-Code-34B (SC, k=50) — 85.1 (2023-09-29) ToRA 70B — 84.3 (2023-09-29) ToRA-Code 34B — 80.7 (2023-09-29) ToRA-Code 13B — 75.8 (2023-09-29) ToRA-Code 7B — 72.6 (2023-09-29) MathCoder-L-70B — 83.9 (2023-10-05) MathCoder-CL-34B — 81.7 (2023-10-05) MathCoder-CL-13B — 74.1 (2023-10-05) MathCoder-L-13B — 72.6 (2023-10-05) MathCoder-CL-7B — 67.8 (2023-10-05) MathCoder-L-7B — 64.2 (2023-10-05) MuggleMATH 70B — 82.3 (2023-10-09) MuggleMATH 13B — 74.0 (2023-10-09) MuggleMATH 7B — 69.8 (2023-10-09) Mistral 7B (maj@8) — 52.2 (2023-10-10) KwaiYiiMath 13B — 73.3 (2023-10-11) Llemma 34B — 51.5 (2023-10-16) Llemma 7B — 36.4 (2023-10-16) ChatGPT (Ask, Refine, Trust) — 82.6 (2023-11-14) OVM-Mistral-7B (verify100@1) — 84.7 (2023-11-16) OVM-Mistral-7B (verify20@1) — 82.6 (2023-11-16) OVM-Llama2-7B (verify100@1) — 73.7 (2023-11-16) Orca 2 13B — 59.14 (2023-11-18) Orca 2 7B — 47.23 (2023-11-18) Vicuna (SYRELM) — 35.2 (2023-12-09) Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) — 89.1 (2023-12-14) Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) — 84.1 (2023-12-14) Phi-GSM+V 1.3B+1.3B (verify48@1) — 81.5 (2023-12-14) Phi-GSM 2.7B (fine-tuned) — 74.3 (2023-12-14) LLaMA 2 70B (CoT-Influx) — 59.59 (2023-12-14) Gemini Pro (maj1@32) — 86.5 (2023-12-19) Camelidae-8×34B (5-shot) — 78.3 (2024-01-05) Qwen2idae-16x14B (5-shot) — 77.8 (2024-01-05) DeepSeekMATH-RL-7B — 88.2 (2024-02-05) Llama-2 70B (on 100 first questions, 4-shot, auto-optimized prompting) — 61.0 (2024-02-09) Llama-2 13B (on 100 first questions, 4-shot, auto-optimized prompting) — 43.0 (2024-02-09) Mistral 7B (on 100 first questions, 4-shot, auto-optimized prompting) — 41.0 (2024-02-09) OpenMath-CodeLlama-70B (w/ code, SC, k=50) — 90.8 (2024-02-15) OpenMath-Llama2-70B (w/ code, SC, k=50) — 90.1 (2024-02-15) OpenMath-CodeLlama-34B (w/ code, SC, k=50) — 88.0 (2024-02-15) OpenMath-Mistral-7B (w/ code, SC, k=50) — 86.9 (2024-02-15) OpenMath-CodeLlama-13B (w/ code, SC, k=50) — 86.8 (2024-02-15) OpenMath-CodeLlama-7B (w/ code, SC, k=50) — 84.8 (2024-02-15) OpenMath-Llama2-70B (w/ code) — 84.7 (2024-02-15) OpenMath-CodeLlama-70B (w/ code) — 84.6 (2024-02-15) OpenMath-CodeLlama-34B (w/ code) — 80.7 (2024-02-15) OpenMath-Mistral-7B (w/ code) — 80.2 (2024-02-15) OpenMath-CodeLlama-13B (w/ code) — 78.8 (2024-02-15) OpenMath-CodeLlama-7B (w/ code) — 75.9 (2024-02-15) Orca-Math 7B (fine-tuned) — 86.8 (2024-02-16) MMOS-DeepSeekMath-7B(0-shot,k=50) — 87.2 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot) — 80.5 (2024-02-23) MMOS-CODE-34B(0-shot) — 80.4 (2024-02-23) MMOS-CODE-7B(0-shot) — 73.9 (2024-02-23) Claude 3 Opus (0-shot chain-of-thought) — 95.0 (2024-03-04) Claude 3 Sonnet (0-shot chain-of-thought) — 92.3 (2024-03-04) Claude 3 Haiku (0-shot chain-of-thought) — 88.9 (2024-03-04) Branch-Train-MiX 4x7B (sampling top-2 experts) — 37.1 (2024-03-12) AlphaLLM (with MCTS) — 92.0 (2024-04-18) DUP prompt upon GPT-4 — 97.1 (2024-04-23) Claude 3.5 Sonnet (HPT) — 97.72 (2024-06-18) GaC(Qwen2-72B-Instruct + Llama-3-70B-Instruct) — 90.91 (2024-06-18) DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) — 90.4 (2024-06-18) DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) — 89.6 (2024-06-18) DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) — 88.2 (2024-06-18) DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) — 86.8 (2024-06-18) DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) — 82.6 (2024-06-18) DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) — 82.5 (2024-06-18) DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) — 81.1 (2024-06-18) DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) — 81.1 (2024-06-18) Qwen2-72B-Instruct-Step-DPO (0-shot CoT) — 94.0 (2024-06-26) Qwen2-Math-72B-Instruct (greedy) — 96.7 (2024-07-15) OpenMath2-Llama3.1-70B (majority@256) — 96.0 (2024-10-02) OpenMath2-Llama3.1-70B — 94.9 (2024-10-02) OpenMath2-Llama3.1-8B (majority@256) — 94.1 (2024-10-02) OpenMath2-Llama3.1-8B — 91.7 (2024-10-02) GPT-4 (Teaching-Inspired) — 94.8 (2024-10-10) Llama-3.3-70B + CAPO — 73.73 (2025-04-22) Mistral-Small-24B + CAPO — 65.07 (2025-04-22) Qwen2.5-32B + CAPO — 60.2 (2025-04-22) Before you <think>, monitor: Implementin — 75.42 (2025-10-18) Testing LLM Arithmetic Reasoning General — 12.16 (2026-06-02) PaLM 540B maj1@40 (8-shot) — 74.4 (2022-03-21) DIVERSE 175B (8-shot) — 83.2 (2022-06-06) Minerva 62B (maj5@100) — 89.0 (2022-06-29) PaLM 2 (few-shot, k=8, SC) — 91.0 (2023-05-17) Claude 3 Opus (0-shot chain-of-thought) — 95.0 (2024-03-04) DUP prompt upon GPT-4 — 97.1 (2024-04-23) Claude 3.5 Sonnet (HPT) — 97.72 (2024-06-18)
RankModel AccuracyParameters (Billion) Extra Training Data PaperCodeYear
1 Claude 3.5 Sonnet (HPT) 97.72– Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles devichand579/HPT 2024
2 DUP prompt upon GPT-4 97.1– Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems whu-zqh/dup 2024
3 Qwen2-Math-72B-Instruct (greedy) 96.772 ✓ Qwen2 Technical Report qwenlm/qwen1.5 · qwenlm/qwen2 · vicentvankor/sun-shine · +3 2024
4 SFT-Mistral-7B (Metamath, OVM, Smart Ensemble) 96.47 ✓
5 OpenMath2-Llama3.1-70B (majority@256) 96.0– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
6 Jiutian-大模型 95.275
7 DAMOMath-7B(MetaMath, OVM, BS, Ensemble) 95.17 ✓
8 Claude 3 Opus (0-shot chain-of-thought) 95– The Claude 3 Model Family: Opus, Sonnet, Haiku 2024
9 OpenMath2-Llama3.1-70B 94.9– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
10 GPT-4 (Teaching-Inspired) 94.8– Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models sallytan13/teaching-inspired-prompting 2024
11 SFT-Mistral-7B (Metamath + ovm +ensemble) 94.137 ✓
12 OpenMath2-Llama3.1-8B (majority@256) 94.1– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
13 Qwen2-72B-Instruct-Step-DPO (0-shot CoT) 94.0– ✓ Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs dvlab-research/step-dpo 2024
14 DAMOMath-7B(MetaMath, OVM, Ensemble) 93.27 ✓
15 Claude 3 Sonnet (0-shot chain-of-thought) 92.3– The Claude 3 Model Family: Opus, Sonnet, Haiku 2024
16 AlphaLLM (with MCTS) 9270 Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing yetianjhu/alphallm 2024
17 OpenMath2-Llama3.1-8B 91.7– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
18 PaLM 2 (few-shot, k=8, SC) 91.0– PaLM 2 Technical Report eternityyw/tram-benchmark 2023
19 GaC(Qwen2-72B-Instruct + Llama-3-70B-Instruct) 90.91– Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling yaoching0/gac 2024
20 OpenMath-CodeLlama-70B (w/ code, SC, k=50) 90.870 ✓ OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset kipok/nemo-skills 2024
1–20 / 166 다음 → 페이지당 10 20 50 100