LLMs cannot spot math errors, even when allowed to peek into the solution
Large language models (LLMs) demonstrate remarkable performance on math word problems, yet they have been shown to struggle with meta-reasoning tasks such as identifying errors in student solutions. In this work, we investigate the challenge of locating the first error step in stepwise solutions using two error reasoning datasets: VtG and PRM800K. Our experiments show that state-of-the-art LLMs struggle to locate the first error step in student solutions even when given access to the reference solution. To that end, we propose an approach that generates an intermediate corrected student solution, aligning more closely with the original student's solution, which helps improve performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Syntactic Blind Spots: How Misalignment Leads to LLMs Mathematical Errors
Large Language Models (LLMs) demonstrate strong mathematical problem-solving abilities but frequently fail on problems that deviate syntactically from their training distribution. We identify a systematic failure mode, s…
Discovering Blind Spots in Reinforcement Learning
Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict the…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
Although large language models (LLMs) have transformed AI, they still make mistakes and can explore unproductive reasoning paths. Self-correction capability is essential for deploying LLMs in safety-critical applications…
Reinforcement LearningESCAPE: Countering Systematic Errors from Machine's Blind Spots via Interactive Visual Analysis
Classification models learn to generalize the associations between data samples and their target classes. However, researchers have increasingly observed that machine learning practice easily leads to systematic errors i…
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for cr…
Misconceptionsscientific discovery