Limits of an AI program for solving college math problems
Drori et al. (2022) report that "A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level ... [It] automatically answers 81\% of university-level mathematics problems." The system they describe is indeed impressive; however, the above description is very much overstated. The work of solving the problems is done, not by a neural network, but by the symbolic algebra package Sympy. Problems of various formats are excluded from consideration. The so-called "explanations" are just rewordings of lines of code. Answers are marked as correct that are not in the form specified in the problem. Most seriously, it seems that in many cases the system uses the correct answer given in the test corpus to guide its path to solving the problem.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningMathProgram SynthesisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains
We introduce FinanceMath, a novel benchmark designed to evaluate LLMs' capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, Fin…
MathMath Word Problem SolvingRetrievalChatGPT-4 with Code Interpreter can be used to solve introductory college-level vector calculus and electromagnetism problems
We evaluated ChatGPT 3.5, 4, and 4 with Code Interpreter on a set of college-level engineering-math and electromagnetism problems, such as those often given to sophomore electrical engineering majors. We selected a set o…
Electrical EngineeringMathMathScale: Scaling Instruction Tuning for Mathematical Reasoning
Large language models (LLMs) have demonstrated remarkable capabilities in problem-solving. However, their proficiency in solving mathematical problems remains inadequate. We propose MathScale, a simple and scalable metho…
GSM8KMathMathematical ReasoningVerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency
Large Language Models (LLMs), combined with program-based solving techniques, are increasingly demonstrating proficiency in mathematical reasoning. For example, closed-source models such as OpenAI GPT-4 and Claude show e…
MathMathematical ReasoningMath Word Problem SolvingSolving Quantitative Reasoning Problems with Language Models
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitati…
Arithmetic ReasoningLanguage ModelingLanguage ModellingLarge Language Model+3