paper-with-me

Math Word Problem Solving 벤치마크

Math Word Problem Solving on MATH

540개 결과 · ⬇ CSV · JSON

Accuracy

1.5 23.15 44.8 66.45 88.1 2021-03 2026-11 GPT-2 (1.5B) — 6.9 (2021-03-05) GPT-2 (0.7B) — 6.4 (2021-03-05) GPT-2 (0.3B) — 6.2 (2021-03-05) GPT-3 13B — 5.6 (2021-03-05) GPT-2 (0.1B) — 5.4 (2021-03-05) GPT-3-175B (few-shot) — 5.2 (2021-03-05) GPT-3-13B (few-shot) — 3.0 (2021-03-05) GPT-3 2.7B — 2.9 (2021-03-05) GPT-2 (1.5B) — 6.9 (2021-03-05) GPT-2 (0.7B) — 6.4 (2021-03-05) GPT-2 (0.3B) — 6.2 (2021-03-05) GPT-3 13B — 5.6 (2021-03-05) GPT-2 (0.1B) — 5.4 (2021-03-05) GPT-3-175B (few-shot) — 5.2 (2021-03-05) GPT-3-13B (few-shot) — 3.0 (2021-03-05) GPT-3 2.7B — 2.9 (2021-03-05) GPT-2 (1.5B) — 6.9 (2021-03-05) GPT-2 (0.7B) — 6.4 (2021-03-05) GPT-2 (0.3B) — 6.2 (2021-03-05) GPT-3 13B — 5.6 (2021-03-05) GPT-2 (0.1B) — 5.4 (2021-03-05) GPT-3-175B (few-shot) — 5.2 (2021-03-05) GPT-3-13B (few-shot) — 3.0 (2021-03-05) GPT-3 2.7B — 2.9 (2021-03-05) GPT-2 (1.5B) — 6.9 (2021-03-05) GPT-2 (0.7B) — 6.4 (2021-03-05) GPT-2 (0.3B) — 6.2 (2021-03-05) GPT-3 13B — 5.6 (2021-03-05) GPT-2 (0.1B) — 5.4 (2021-03-05) GPT-3-175B (few-shot) — 5.2 (2021-03-05) GPT-3-13B (few-shot) — 3.0 (2021-03-05) GPT-3 2.7B — 2.9 (2021-03-05) Minerva 62B (maj5@256) — 64.9 (2022-06-29) Minerva 540B (maj1@k, k=64) — 50.3 (2022-06-29) Minerva 8B (maj5@256) — 47.6 (2022-06-29) Minerva 62B (maj1@k, k=64) — 43.4 (2022-06-29) Minerva 540B — 33.6 (2022-06-29) Minerva 62B (4-shot) — 27.6 (2022-06-29) Minerva 8B (maj1@k, k=64) — 25.4 (2022-06-29) davinci-002 175B — 19.1 (2022-06-29) Minerva 8B — 14.1 (2022-06-29) PaLM 540B — 8.8 (2022-06-29) PaLM 8B (fine-tuned) — 5.6 (2022-06-29) PaLM 62B — 4.4 (2022-06-29) PaLM 8B — 1.5 (2022-06-29) Minerva 62B (maj5@256) — 64.9 (2022-06-29) Minerva 540B (maj1@k, k=64) — 50.3 (2022-06-29) Minerva 8B (maj5@256) — 47.6 (2022-06-29) Minerva 62B (maj1@k, k=64) — 43.4 (2022-06-29) Minerva 540B — 33.6 (2022-06-29) Minerva 62B (4-shot) — 27.6 (2022-06-29) Minerva 8B (maj1@k, k=64) — 25.4 (2022-06-29) davinci-002 175B — 19.1 (2022-06-29) Minerva 8B — 14.1 (2022-06-29) PaLM 540B — 8.8 (2022-06-29) PaLM 8B (fine-tuned) — 5.6 (2022-06-29) PaLM 62B — 4.4 (2022-06-29) PaLM 8B — 1.5 (2022-06-29) Minerva 62B (maj5@256) — 64.9 (2022-06-29) Minerva 540B (maj1@k, k=64) — 50.3 (2022-06-29) Minerva 8B (maj5@256) — 47.6 (2022-06-29) Minerva 62B (maj1@k, k=64) — 43.4 (2022-06-29) Minerva 540B — 33.6 (2022-06-29) Minerva 62B (4-shot) — 27.6 (2022-06-29) Minerva 8B (maj1@k, k=64) — 25.4 (2022-06-29) davinci-002 175B — 19.1 (2022-06-29) Minerva 8B — 14.1 (2022-06-29) PaLM 540B — 8.8 (2022-06-29) PaLM 8B (fine-tuned) — 5.6 (2022-06-29) PaLM 62B — 4.4 (2022-06-29) PaLM 8B — 1.5 (2022-06-29) Minerva 62B (maj5@256) — 64.9 (2022-06-29) Minerva 540B (maj1@k, k=64) — 50.3 (2022-06-29) Minerva 8B (maj5@256) — 47.6 (2022-06-29) Minerva 62B (maj1@k, k=64) — 43.4 (2022-06-29) Minerva 540B — 33.6 (2022-06-29) Minerva 62B (4-shot) — 27.6 (2022-06-29) Minerva 8B (maj1@k, k=64) — 25.4 (2022-06-29) davinci-002 175B — 19.1 (2022-06-29) Minerva 8B — 14.1 (2022-06-29) PaLM 540B — 8.8 (2022-06-29) PaLM 8B (fine-tuned) — 5.6 (2022-06-29) PaLM 62B — 4.4 (2022-06-29) PaLM 8B — 1.5 (2022-06-29) Minerva 540B (5-shot) mCoT — 33.6 (2022-11-16) GAL 120B (5-shot) mCoT — 20.4 (2022-11-16) GAL 120B <work> — 16.6 (2022-11-16) GAL 30B (5-shot) mCoT — 12.7 (2022-11-16) GAL 30B <work> — 11.4 (2022-11-16) PaLM 540B (5-shot) mCoT — 8.8 (2022-11-16) GPT-3 175B (8-shot) — 5.2 (2022-11-16) Minerva 540B (5-shot) mCoT — 33.6 (2022-11-16) GAL 120B (5-shot) mCoT — 20.4 (2022-11-16) GAL 120B <work> — 16.6 (2022-11-16) GAL 30B (5-shot) mCoT — 12.7 (2022-11-16) GAL 30B <work> — 11.4 (2022-11-16) PaLM 540B (5-shot) mCoT — 8.8 (2022-11-16) GPT-3 175B (8-shot) — 5.2 (2022-11-16) Minerva 540B (5-shot) mCoT — 33.6 (2022-11-16) GAL 120B (5-shot) mCoT — 20.4 (2022-11-16) GAL 120B <work> — 16.6 (2022-11-16) GAL 30B (5-shot) mCoT — 12.7 (2022-11-16) GAL 30B <work> — 11.4 (2022-11-16) PaLM 540B (5-shot) mCoT — 8.8 (2022-11-16) GPT-3 175B (8-shot) — 5.2 (2022-11-16) Minerva 540B (5-shot) mCoT — 33.6 (2022-11-16) GAL 120B (5-shot) mCoT — 20.4 (2022-11-16) GAL 120B <work> — 16.6 (2022-11-16) GAL 30B (5-shot) mCoT — 12.7 (2022-11-16) GAL 30B <work> — 11.4 (2022-11-16) PaLM 540B (5-shot) mCoT — 8.8 (2022-11-16) GPT-3 175B (8-shot) — 5.2 (2022-11-16) GPT-4 model (w/ code, PAL) — 51.8 (2022-11-18) GPT-4 model (w/ code, PAL) — 51.8 (2022-11-18) GPT-4 model (w/ code, PAL) — 51.8 (2022-11-18) GPT-4 model (w/ code, PAL) — 51.8 (2022-11-18) LLaMA 65B (maj1@k) — 20.5 (2023-02-27) LLaMA 33B-maj1@k — 15.2 (2023-02-27) LLaMA 65B — 10.6 (2023-02-27) LLaMA 13B-maj1@k — 8.8 (2023-02-27) LLaMA 33B — 7.1 (2023-02-27) LLaMA 7B-maj1@k — 6.9 (2023-02-27) LLaMA 13B — 3.9 (2023-02-27) LLaMA 7B — 2.9 (2023-02-27) LLaMA 65B (maj1@k) — 20.5 (2023-02-27) LLaMA 33B-maj1@k — 15.2 (2023-02-27) LLaMA 65B — 10.6 (2023-02-27) LLaMA 13B-maj1@k — 8.8 (2023-02-27) LLaMA 33B — 7.1 (2023-02-27) LLaMA 7B-maj1@k — 6.9 (2023-02-27) LLaMA 13B — 3.9 (2023-02-27) LLaMA 7B — 2.9 (2023-02-27) LLaMA 65B (maj1@k) — 20.5 (2023-02-27) LLaMA 33B-maj1@k — 15.2 (2023-02-27) LLaMA 65B — 10.6 (2023-02-27) LLaMA 13B-maj1@k — 8.8 (2023-02-27) LLaMA 33B — 7.1 (2023-02-27) LLaMA 7B-maj1@k — 6.9 (2023-02-27) LLaMA 13B — 3.9 (2023-02-27) LLaMA 7B — 2.9 (2023-02-27) LLaMA 65B (maj1@k) — 20.5 (2023-02-27) LLaMA 33B-maj1@k — 15.2 (2023-02-27) LLaMA 65B — 10.6 (2023-02-27) LLaMA 13B-maj1@k — 8.8 (2023-02-27) LLaMA 33B — 7.1 (2023-02-27) LLaMA 7B-maj1@k — 6.9 (2023-02-27) LLaMA 13B — 3.9 (2023-02-27) LLaMA 7B — 2.9 (2023-02-27) GPT-4 — 42.5 (2023-03-22) GPT-4 — 42.5 (2023-03-22) GPT-4 — 42.5 (2023-03-22) GPT-4 — 42.5 (2023-03-22) PHP (GPT-4 model) — 53.9 (2023-04-19) PHP (GPT-4 model) — 53.9 (2023-04-19) PHP (GPT-4 model) — 53.9 (2023-04-19) PHP (GPT-4 model) — 53.9 (2023-04-19) PaLM 2 (few-shot, k=4, SC) — 48.8 (2023-05-17) PaLM 2 (few-shot, k=4, CoT) — 34.3 (2023-05-17) PaLM 2 (few-shot, k=4, SC) — 48.8 (2023-05-17) PaLM 2 (few-shot, k=4, CoT) — 34.3 (2023-05-17) PaLM 2 (few-shot, k=4, SC) — 48.8 (2023-05-17) PaLM 2 (few-shot, k=4, CoT) — 34.3 (2023-05-17) PaLM 2 (few-shot, k=4, SC) — 48.8 (2023-05-17) PaLM 2 (few-shot, k=4, CoT) — 34.3 (2023-05-17) SKiC (GPT-4 model) — 56.4 (2023-08-01) SKiC (GPT-4 model) — 56.4 (2023-08-01) SKiC (GPT-4 model) — 56.4 (2023-08-01) SKiC (GPT-4 model) — 56.4 (2023-08-01) CR (GPT-4-turbo model, w/ code) — 72.2 (2023-08-08) CR (GPT-4 model, w/o code) — 58.0 (2023-08-08) CR (GPT-4-turbo model, w/ code) — 72.2 (2023-08-08) CR (GPT-4 model, w/o code) — 58.0 (2023-08-08) CR (GPT-4-turbo model, w/ code) — 72.2 (2023-08-08) CR (GPT-4 model, w/o code) — 58.0 (2023-08-08) CR (GPT-4-turbo model, w/ code) — 72.2 (2023-08-08) CR (GPT-4 model, w/o code) — 58.0 (2023-08-08) GPT-4-code model (CSV, w/ code, SC, k=16) — 84.3 (2023-08-15) GPT-4-code model (CSV, w/ code) — 73.5 (2023-08-15) LogicNet (with code interpreter) — 71.2 (2023-08-15) GPT-4-code model (w/ code) — 69.7 (2023-08-15) GPT-4-code model (w/o code) — 60.8 (2023-08-15) GPT-4-code model (CSV, w/ code, SC, k=16) — 84.3 (2023-08-15) GPT-4-code model (CSV, w/ code) — 73.5 (2023-08-15) LogicNet (with code interpreter) — 71.2 (2023-08-15) GPT-4-code model (w/ code) — 69.7 (2023-08-15) GPT-4-code model (w/o code) — 60.8 (2023-08-15) GPT-4-code model (CSV, w/ code, SC, k=16) — 84.3 (2023-08-15) GPT-4-code model (CSV, w/ code) — 73.5 (2023-08-15) LogicNet (with code interpreter) — 71.2 (2023-08-15) GPT-4-code model (w/ code) — 69.7 (2023-08-15) GPT-4-code model (w/o code) — 60.8 (2023-08-15) GPT-4-code model (CSV, w/ code, SC, k=16) — 84.3 (2023-08-15) GPT-4-code model (CSV, w/ code) — 73.5 (2023-08-15) LogicNet (with code interpreter) — 71.2 (2023-08-15) GPT-4-code model (w/ code) — 69.7 (2023-08-15) GPT-4-code model (w/o code) — 60.8 (2023-08-15) WizardMath-7B-V1.1 — 33.0 (2023-08-18) WizardMath-70B-V1.0 — 22.7 (2023-08-18) WizardMath-13B-V1.0 — 14.0 (2023-08-18) WizardMath-7B-V1.0 — 10.7 (2023-08-18) WizardMath-7B-V1.1 — 33.0 (2023-08-18) WizardMath-70B-V1.0 — 22.7 (2023-08-18) WizardMath-13B-V1.0 — 14.0 (2023-08-18) WizardMath-7B-V1.0 — 10.7 (2023-08-18) WizardMath-7B-V1.1 — 33.0 (2023-08-18) WizardMath-70B-V1.0 — 22.7 (2023-08-18) WizardMath-13B-V1.0 — 14.0 (2023-08-18) WizardMath-7B-V1.0 — 10.7 (2023-08-18) WizardMath-7B-V1.1 — 33.0 (2023-08-18) WizardMath-70B-V1.0 — 22.7 (2023-08-18) WizardMath-13B-V1.0 — 14.0 (2023-08-18) WizardMath-7B-V1.0 — 10.7 (2023-08-18) OpenChat-3.5-1210 7B — 28.9 (2023-09-20) OpenChat-3.5 7B — 28.6 (2023-09-20) OpenChat-3.5-1210 7B — 28.9 (2023-09-20) OpenChat-3.5 7B — 28.6 (2023-09-20) OpenChat-3.5-1210 7B — 28.9 (2023-09-20) OpenChat-3.5 7B — 28.6 (2023-09-20) OpenChat-3.5-1210 7B — 28.9 (2023-09-20) OpenChat-3.5 7B — 28.6 (2023-09-20) MetaMath 70B — 26.0 (2023-09-21) MetaMath 13B — 22.5 (2023-09-21) MetaMath 7B — 19.4 (2023-09-21) MetaMath 70B — 26.0 (2023-09-21) MetaMath 13B — 22.5 (2023-09-21) MetaMath 7B — 19.4 (2023-09-21) MetaMath 70B — 26.0 (2023-09-21) MetaMath 13B — 22.5 (2023-09-21) MetaMath 7B — 19.4 (2023-09-21) MetaMath 70B — 26.0 (2023-09-21) MetaMath 13B — 22.5 (2023-09-21) MetaMath 7B — 19.4 (2023-09-21) ToRA-Code 34B model (w/ code, SC, k=50) — 60.0 (2023-09-29) ToRA 70B (w/ code, SC, k=50) — 56.9 (2023-09-29) ToRA-Code 34B (w/ code) — 50.8 (2023-09-29) ToRA 70B (w/ code) — 49.7 (2023-09-29) ToRA-Code 13B (w/ code) — 48.1 (2023-09-29) ToRA-Code 7B (w/ code) — 44.6 (2023-09-29) ToRA 13B (w/ code) — 43.0 (2023-09-29) ToRA 7B (w/ code) — 40.1 (2023-09-29) ToRA-Code 34B model (w/ code, SC, k=50) — 60.0 (2023-09-29) ToRA 70B (w/ code, SC, k=50) — 56.9 (2023-09-29) ToRA-Code 34B (w/ code) — 50.8 (2023-09-29) ToRA 70B (w/ code) — 49.7 (2023-09-29) ToRA-Code 13B (w/ code) — 48.1 (2023-09-29) ToRA-Code 7B (w/ code) — 44.6 (2023-09-29) ToRA 13B (w/ code) — 43.0 (2023-09-29) ToRA 7B (w/ code) — 40.1 (2023-09-29) ToRA-Code 34B model (w/ code, SC, k=50) — 60.0 (2023-09-29) ToRA 70B (w/ code, SC, k=50) — 56.9 (2023-09-29) ToRA-Code 34B (w/ code) — 50.8 (2023-09-29) ToRA 70B (w/ code) — 49.7 (2023-09-29) ToRA-Code 13B (w/ code) — 48.1 (2023-09-29) ToRA-Code 7B (w/ code) — 44.6 (2023-09-29) ToRA 13B (w/ code) — 43.0 (2023-09-29) ToRA 7B (w/ code) — 40.1 (2023-09-29) ToRA-Code 34B model (w/ code, SC, k=50) — 60.0 (2023-09-29) ToRA 70B (w/ code, SC, k=50) — 56.9 (2023-09-29) ToRA-Code 34B (w/ code) — 50.8 (2023-09-29) ToRA 70B (w/ code) — 49.7 (2023-09-29) ToRA-Code 13B (w/ code) — 48.1 (2023-09-29) ToRA-Code 7B (w/ code) — 44.6 (2023-09-29) ToRA 13B (w/ code) — 43.0 (2023-09-29) ToRA 7B (w/ code) — 40.1 (2023-09-29) MathCoder-CL-34B — 45.2 (2023-10-05) MathCoder-L-34B — 45.1 (2023-10-05) MathCoder-CL-13B — 35.9 (2023-10-05) MathCoder-CL-7B — 30.2 (2023-10-05) MathCoder-L-13B — 29.9 (2023-10-05) MathCoder-L-7B — 23.3 (2023-10-05) MathCoder-CL-34B — 45.2 (2023-10-05) MathCoder-L-34B — 45.1 (2023-10-05) MathCoder-CL-13B — 35.9 (2023-10-05) MathCoder-CL-7B — 30.2 (2023-10-05) MathCoder-L-13B — 29.9 (2023-10-05) MathCoder-L-7B — 23.3 (2023-10-05) MathCoder-CL-34B — 45.2 (2023-10-05) MathCoder-L-34B — 45.1 (2023-10-05) MathCoder-CL-13B — 35.9 (2023-10-05) MathCoder-CL-7B — 30.2 (2023-10-05) MathCoder-L-13B — 29.9 (2023-10-05) MathCoder-L-7B — 23.3 (2023-10-05) MathCoder-CL-34B — 45.2 (2023-10-05) MathCoder-L-34B — 45.1 (2023-10-05) MathCoder-CL-13B — 35.9 (2023-10-05) MathCoder-CL-7B — 30.2 (2023-10-05) MathCoder-L-13B — 29.9 (2023-10-05) MathCoder-L-7B — 23.3 (2023-10-05) MuggleMATH-70B — 35.6 (2023-10-09) MuggleMATH-13B — 30.7 (2023-10-09) MuggleMATH 7B — 25.8 (2023-10-09) MuggleMATH-70B — 35.6 (2023-10-09) MuggleMATH-13B — 30.7 (2023-10-09) MuggleMATH 7B — 25.8 (2023-10-09) MuggleMATH-70B — 35.6 (2023-10-09) MuggleMATH-13B — 30.7 (2023-10-09) MuggleMATH 7B — 25.8 (2023-10-09) MuggleMATH-70B — 35.6 (2023-10-09) MuggleMATH-13B — 30.7 (2023-10-09) MuggleMATH 7B — 25.8 (2023-10-09) Mistral 7B (maj@4) — 13.1 (2023-10-10) Mistral 7B (maj@4) — 13.1 (2023-10-10) Mistral 7B (maj@4) — 13.1 (2023-10-10) Mistral 7B (maj@4) — 13.1 (2023-10-10) Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) — 48.1 (2023-12-14) Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) — 43.5 (2023-12-14) Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) — 33.0 (2023-12-14) Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) — 48.1 (2023-12-14) Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) — 43.5 (2023-12-14) Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) — 33.0 (2023-12-14) Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) — 48.1 (2023-12-14) Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) — 43.5 (2023-12-14) Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) — 33.0 (2023-12-14) Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) — 48.1 (2023-12-14) Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) — 43.5 (2023-12-14) Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) — 33.0 (2023-12-14) Gemini Ultra (4-shot) — 53.2 (2023-12-19) Gemini Pro (4-shot) — 32.6 (2023-12-19) Gemini Ultra (4-shot) — 53.2 (2023-12-19) Gemini Pro (4-shot) — 32.6 (2023-12-19) Gemini Ultra (4-shot) — 53.2 (2023-12-19) Gemini Pro (4-shot) — 32.6 (2023-12-19) Gemini Ultra (4-shot) — 53.2 (2023-12-19) Gemini Pro (4-shot) — 32.6 (2023-12-19) Qwen2idae-16x14B (4-shot) — 29.9 (2024-01-05) Camelidae-8×34B (4-shot) — 22.6 (2024-01-05) Qwen2idae-16x14B (4-shot) — 29.9 (2024-01-05) Camelidae-8×34B (4-shot) — 22.6 (2024-01-05) Qwen2idae-16x14B (4-shot) — 29.9 (2024-01-05) Camelidae-8×34B (4-shot) — 22.6 (2024-01-05) Qwen2idae-16x14B (4-shot) — 29.9 (2024-01-05) Camelidae-8×34B (4-shot) — 22.6 (2024-01-05) Mixtral 8x7B (maj@4) — 28.4 (2024-01-08) Mistral 7B (maj@4) — 12.7 (2024-01-08) Mixtral 8x7B (maj@4) — 28.4 (2024-01-08) Mistral 7B (maj@4) — 12.7 (2024-01-08) Mixtral 8x7B (maj@4) — 28.4 (2024-01-08) Mistral 7B (maj@4) — 12.7 (2024-01-08) Mixtral 8x7B (maj@4) — 28.4 (2024-01-08) Mistral 7B (maj@4) — 12.7 (2024-01-08) MMIQC-72B — 45.0 (2024-01-17) MMIQC-72B — 45.0 (2024-01-17) MMIQC-72B — 45.0 (2024-01-17) MMIQC-72B — 45.0 (2024-01-17) DeepSeekMATH-RL-7B (w/ code, greedy decoding) — 58.8 (2024-02-05) DeepSeekMATH-RL-7B (greedy decoding) — 51.7 (2024-02-05) DeepSeekMATH-RL-7B (w/ code, greedy decoding) — 58.8 (2024-02-05) DeepSeekMATH-RL-7B (greedy decoding) — 51.7 (2024-02-05) DeepSeekMATH-RL-7B (w/ code, greedy decoding) — 58.8 (2024-02-05) DeepSeekMATH-RL-7B (greedy decoding) — 51.7 (2024-02-05) DeepSeekMATH-RL-7B (w/ code, greedy decoding) — 58.8 (2024-02-05) DeepSeekMATH-RL-7B (greedy decoding) — 51.7 (2024-02-05) OpenMath-CodeLlama-70B (w/ code, SC, k=50) — 60.4 (2024-02-15) OpenMath-CodeLlama-34B (w/ code, SC, k=50) — 60.2 (2024-02-15) OpenMath-Llama2-70B (w/ code, SC, k=50) — 58.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code, SC, k=50) — 57.6 (2024-02-15) OpenMath-Mistral-7B (w/ code, SC, k=50) — 57.2 (2024-02-15) OpenMath-CodeLlama-7B (w/ code, SC, k=50) — 55.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code) — 50.7 (2024-02-15) OpenMath-CodeLlama-34B (w/ code) — 48.3 (2024-02-15) OpenMath-Llama2-70B (w/ code) — 46.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code) — 45.5 (2024-02-15) OpenMath-Mistral-7B (w/ code) — 44.5 (2024-02-15) OpenMath-CodeLlama-7B (w/ code) — 43.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code, SC, k=50) — 60.4 (2024-02-15) OpenMath-CodeLlama-34B (w/ code, SC, k=50) — 60.2 (2024-02-15) OpenMath-Llama2-70B (w/ code, SC, k=50) — 58.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code, SC, k=50) — 57.6 (2024-02-15) OpenMath-Mistral-7B (w/ code, SC, k=50) — 57.2 (2024-02-15) OpenMath-CodeLlama-7B (w/ code, SC, k=50) — 55.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code) — 50.7 (2024-02-15) OpenMath-CodeLlama-34B (w/ code) — 48.3 (2024-02-15) OpenMath-Llama2-70B (w/ code) — 46.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code) — 45.5 (2024-02-15) OpenMath-Mistral-7B (w/ code) — 44.5 (2024-02-15) OpenMath-CodeLlama-7B (w/ code) — 43.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code, SC, k=50) — 60.4 (2024-02-15) OpenMath-CodeLlama-34B (w/ code, SC, k=50) — 60.2 (2024-02-15) OpenMath-Llama2-70B (w/ code, SC, k=50) — 58.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code, SC, k=50) — 57.6 (2024-02-15) OpenMath-Mistral-7B (w/ code, SC, k=50) — 57.2 (2024-02-15) OpenMath-CodeLlama-7B (w/ code, SC, k=50) — 55.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code) — 50.7 (2024-02-15) OpenMath-CodeLlama-34B (w/ code) — 48.3 (2024-02-15) OpenMath-Llama2-70B (w/ code) — 46.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code) — 45.5 (2024-02-15) OpenMath-Mistral-7B (w/ code) — 44.5 (2024-02-15) OpenMath-CodeLlama-7B (w/ code) — 43.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code, SC, k=50) — 60.4 (2024-02-15) OpenMath-CodeLlama-34B (w/ code, SC, k=50) — 60.2 (2024-02-15) OpenMath-Llama2-70B (w/ code, SC, k=50) — 58.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code, SC, k=50) — 57.6 (2024-02-15) OpenMath-Mistral-7B (w/ code, SC, k=50) — 57.2 (2024-02-15) OpenMath-CodeLlama-7B (w/ code, SC, k=50) — 55.6 (2024-02-15) OpenMath-CodeLlama-70B (w/ code) — 50.7 (2024-02-15) OpenMath-CodeLlama-34B (w/ code) — 48.3 (2024-02-15) OpenMath-Llama2-70B (w/ code) — 46.3 (2024-02-15) OpenMath-CodeLlama-13B (w/ code) — 45.5 (2024-02-15) OpenMath-Mistral-7B (w/ code) — 44.5 (2024-02-15) OpenMath-CodeLlama-7B (w/ code) — 43.6 (2024-02-15) MMOS-DeepSeekMath-7B(0-shot,k=50) — 63.7 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot) — 55.0 (2024-02-23) MMOS-CODE-34B(0-shot) — 49.5 (2024-02-23) MMOS-CODE-7B(0-shot) — 44.3 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot,k=50) — 63.7 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot) — 55.0 (2024-02-23) MMOS-CODE-34B(0-shot) — 49.5 (2024-02-23) MMOS-CODE-7B(0-shot) — 44.3 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot,k=50) — 63.7 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot) — 55.0 (2024-02-23) MMOS-CODE-34B(0-shot) — 49.5 (2024-02-23) MMOS-CODE-7B(0-shot) — 44.3 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot,k=50) — 63.7 (2024-02-23) MMOS-DeepSeekMath-7B(0-shot) — 55.0 (2024-02-23) MMOS-CODE-34B(0-shot) — 49.5 (2024-02-23) MMOS-CODE-7B(0-shot) — 44.3 (2024-02-23) DeepSeekMath-7B-KPMath-Plus — 48.8 (2024-03-04) Llemma-34B-KPMath-Plus — 48.6 (2024-03-04) Mistral-7B-KPMath-Plus — 46.8 (2024-03-04) Llama2-13B-KPMath-Plus — 41.0 (2024-03-04) DeepSeekMath-7B-KPMath-Plus — 48.8 (2024-03-04) Llemma-34B-KPMath-Plus — 48.6 (2024-03-04) Mistral-7B-KPMath-Plus — 46.8 (2024-03-04) Llama2-13B-KPMath-Plus — 41.0 (2024-03-04) DeepSeekMath-7B-KPMath-Plus — 48.8 (2024-03-04) Llemma-34B-KPMath-Plus — 48.6 (2024-03-04) Mistral-7B-KPMath-Plus — 46.8 (2024-03-04) Llama2-13B-KPMath-Plus — 41.0 (2024-03-04) DeepSeekMath-7B-KPMath-Plus — 48.8 (2024-03-04) Llemma-34B-KPMath-Plus — 48.6 (2024-03-04) Mistral-7B-KPMath-Plus — 46.8 (2024-03-04) Llama2-13B-KPMath-Plus — 41.0 (2024-03-04) Branch-Train-MiX 4x7B (sampling top-2 experts) — 17.8 (2024-03-12) Branch-Train-MiX 4x7B (sampling top-2 experts) — 17.8 (2024-03-12) Branch-Train-MiX 4x7B (sampling top-2 experts) — 17.8 (2024-03-12) Branch-Train-MiX 4x7B (sampling top-2 experts) — 17.8 (2024-03-12) GPT-4 Turbo (MACM, w/code, voting) — 87.92 (2024-04-06) GPT-4 Turbo (MACM, w/code, voting) — 87.92 (2024-04-06) GPT-4 Turbo (MACM, w/code, voting) — 87.92 (2024-04-06) GPT-4 Turbo (MACM, w/code, voting) — 87.92 (2024-04-06) AlphaLLM (with MCTS) — 51.0 (2024-04-18) AlphaLLM (with MCTS) — 51.0 (2024-04-18) AlphaLLM (with MCTS) — 51.0 (2024-04-18) AlphaLLM (with MCTS) — 51.0 (2024-04-18) AlphaMath-7B-SBS@3 — 66.3 (2024-05-06) AlphaMath-7B-SBS@3 — 66.3 (2024-05-06) AlphaMath-7B-SBS@3 — 66.3 (2024-05-06) AlphaMath-7B-SBS@3 — 66.3 (2024-05-06) DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) — 56.1 (2024-06-18) DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) — 54.9 (2024-06-18) DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) — 53.6 (2024-06-18) DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) — 52.9 (2024-06-18) DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) — 46.6 (2024-06-18) DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) — 45.5 (2024-06-18) DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) — 45.3 (2024-06-18) DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) — 43.5 (2024-06-18) DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) — 56.1 (2024-06-18) DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) — 54.9 (2024-06-18) DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) — 53.6 (2024-06-18) DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) — 52.9 (2024-06-18) DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) — 46.6 (2024-06-18) DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) — 45.5 (2024-06-18) DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) — 45.3 (2024-06-18) DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) — 43.5 (2024-06-18) DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) — 56.1 (2024-06-18) DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) — 54.9 (2024-06-18) DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) — 53.6 (2024-06-18) DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) — 52.9 (2024-06-18) DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) — 46.6 (2024-06-18) DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) — 45.5 (2024-06-18) DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) — 45.3 (2024-06-18) DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) — 43.5 (2024-06-18) DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) — 56.1 (2024-06-18) DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) — 54.9 (2024-06-18) DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) — 53.6 (2024-06-18) DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) — 52.9 (2024-06-18) DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) — 46.6 (2024-06-18) DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) — 45.5 (2024-06-18) DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) — 45.3 (2024-06-18) DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) — 43.5 (2024-06-18) Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) — 70.8 (2024-06-26) Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) — 70.8 (2024-06-26) Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) — 70.8 (2024-06-26) Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) — 70.8 (2024-06-26) Qwen2-Math-72B-Instruct(greedy) — 84.0 (2024-07-15) Qwen2-Math-72B-Instruct(greedy) — 84.0 (2024-07-15) Qwen2-Math-72B-Instruct(greedy) — 84.0 (2024-07-15) Qwen2-Math-72B-Instruct(greedy) — 84.0 (2024-07-15) Qwen2.5-Math-72B-Instruct(TIR,Greedy) — 88.1 (2024-09-18) Qwen2.5-Math-72B-Instruct(COT,Greedy) — 85.9 (2024-09-18) Qwen2.5-Math-7B-Instruct(TIR,Greedy) — 85.2 (2024-09-18) Qwen2.5-Math-7B-Instruct(COT,Greedy) — 83.6 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) — 79.9 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(COT,Greedy) — 75.8 (2024-09-18) Qwen2.5-Math-72B-Instruct(TIR,Greedy) — 88.1 (2024-09-18) Qwen2.5-Math-72B-Instruct(COT,Greedy) — 85.9 (2024-09-18) Qwen2.5-Math-7B-Instruct(TIR,Greedy) — 85.2 (2024-09-18) Qwen2.5-Math-7B-Instruct(COT,Greedy) — 83.6 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) — 79.9 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(COT,Greedy) — 75.8 (2024-09-18) Qwen2.5-Math-72B-Instruct(TIR,Greedy) — 88.1 (2024-09-18) Qwen2.5-Math-72B-Instruct(COT,Greedy) — 85.9 (2024-09-18) Qwen2.5-Math-7B-Instruct(TIR,Greedy) — 85.2 (2024-09-18) Qwen2.5-Math-7B-Instruct(COT,Greedy) — 83.6 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) — 79.9 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(COT,Greedy) — 75.8 (2024-09-18) Qwen2.5-Math-72B-Instruct(TIR,Greedy) — 88.1 (2024-09-18) Qwen2.5-Math-72B-Instruct(COT,Greedy) — 85.9 (2024-09-18) Qwen2.5-Math-7B-Instruct(TIR,Greedy) — 85.2 (2024-09-18) Qwen2.5-Math-7B-Instruct(COT,Greedy) — 83.6 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) — 79.9 (2024-09-18) Qwen2.5-Math-1.5B-Instruct(COT,Greedy) — 75.8 (2024-09-18) OpenMath2-Llama3.1-70B (majority@256) — 79.6 (2024-10-02) OpenMath2-Llama3.1-8B (majority@256) — 76.1 (2024-10-02) OpenMath2-Llama3.1-70B — 71.9 (2024-10-02) OpenMath2-Llama3.1-8B — 67.8 (2024-10-02) OpenMath2-Llama3.1-70B (majority@256) — 79.6 (2024-10-02) OpenMath2-Llama3.1-8B (majority@256) — 76.1 (2024-10-02) OpenMath2-Llama3.1-70B — 71.9 (2024-10-02) OpenMath2-Llama3.1-8B — 67.8 (2024-10-02) OpenMath2-Llama3.1-70B (majority@256) — 79.6 (2024-10-02) OpenMath2-Llama3.1-8B (majority@256) — 76.1 (2024-10-02) OpenMath2-Llama3.1-70B — 71.9 (2024-10-02) OpenMath2-Llama3.1-8B — 67.8 (2024-10-02) OpenMath2-Llama3.1-70B (majority@256) — 79.6 (2024-10-02) OpenMath2-Llama3.1-8B (majority@256) — 76.1 (2024-10-02) OpenMath2-Llama3.1-70B — 71.9 (2024-10-02) OpenMath2-Llama3.1-8B — 67.8 (2024-10-02) GPT-2 (1.5B) — 6.9 (2021-03-05) Minerva 62B (maj5@256) — 64.9 (2022-06-29) CR (GPT-4-turbo model, w/ code) — 72.2 (2023-08-08) GPT-4-code model (CSV, w/ code, SC, k=16) — 84.3 (2023-08-15) GPT-4 Turbo (MACM, w/code, voting) — 87.92 (2024-04-06) Qwen2.5-Math-72B-Instruct(TIR,Greedy) — 88.1 (2024-09-18)
RankModel AccuracyParameters (Billions) Extra Training Data PaperCodeYear
1 Gemini 2.0 Flash Experimental 89.7–
2 Qwen2.5-Math-72B-Instruct(TIR,Greedy) 88.172 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
3 GPT-4 Turbo (MACM, w/code, voting) 87.920– MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems bin123apple/macm 2024
4 Qwen2.5-Math-72B-Instruct(COT,Greedy) 85.972 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
5 Qwen2.5-Math-7B-Instruct(TIR,Greedy) 85.27 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
6 GPT-4-code model (CSV, w/ code, SC, k=16) 84.3– Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification kipok/nemo-skills 2023
7 Qwen2-Math-72B-Instruct(greedy) 84.072 ✓ Qwen2 Technical Report qwenlm/qwen1.5 · qwenlm/qwen2 · vicentvankor/sun-shine · +3 2024
8 Qwen2.5-Math-7B-Instruct(COT,Greedy) 83.67 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
9 Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) 79.91.5 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
10 OpenMath2-Llama3.1-70B (majority@256) 79.6– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
11 OpenMath2-Llama3.1-8B (majority@256) 76.1– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
12 Qwen2.5-Math-1.5B-Instruct(COT,Greedy) 75.81.5 ✓ Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 2024
13 GPT-4-code model (CSV, w/ code) 73.5– Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification kipok/nemo-skills 2023
14 CR (GPT-4-turbo model, w/ code) 72.2– Cumulative Reasoning with Large Language Models iiis-ai/cumulative-reasoning 2023
15 OpenMath2-Llama3.1-70B 71.9– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
16 LogicNet (with code interpreter) 71.2– ✓ Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification kipok/nemo-skills 2023
17 Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) 70.8– ✓ Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs dvlab-research/step-dpo 2024
18 GPT-4-code model (w/ code) 69.7– Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification kipok/nemo-skills 2023
19 OpenMath2-Llama3.1-8B 67.8– ✓ OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data NVIDIA/NeMo-Skills 2024
20 AlphaMath-7B-SBS@3 66.3– AlphaMath Almost Zero: Process Supervision without Process MARIO-Math-Reasoning/Super_MARIO 2024
1–20 / 540 다음 → 페이지당 10 20 50 100