SMART: Self-Generating and Self-Validating Multi-Dimensional Assessment for LLMs' Mathematical Problem Solving
Large Language Models have achieved remarkable results on a variety of mathematical benchmarks. However, concerns remain as to whether these successes reflect genuine mathematical reasoning or superficial pattern recognition. Common evaluation metrics, such as final answer accuracy, fail to disentangle the underlying competencies involved, offering limited diagnostic value. To address these limitations, we introduce SMART: a Self-Generating and Self-Validating Multi-Dimensional Assessment Framework. SMART decomposes mathematical problem solving into four distinct dimensions: understanding, reasoning, arithmetic, and reflection \& refinement. Each dimension is evaluated independently through tailored tasks, enabling interpretable and fine-grained analysis of LLM behavior. Crucially, SMART integrates an automated self-generating and self-validating mechanism to produce and verify benchmark data, ensuring both scalability and reliability. We apply SMART to 21 state-of-the-art open- and closed-source LLMs, uncovering significant discrepancies in their abilities across different dimensions. Our findings demonstrate the inadequacy of final answer accuracy as a sole metric and motivate a new holistic metric to better capture true problem-solving capabilities. Code and benchmarks will be released upon acceptance.
Code (0)
등록된 구현이 없습니다.
Tasks
DiagnosticMathematical Problem-SolvingMathematical ReasoningSimilar Papers 제목 키워드 기반
ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation
Vocabulary acquisition is essential to second language learning, as it underpins all core language skills. Accurate vocabulary assessment is particularly important in standardized exams, where test items evaluate learner…
Distractor GenerationvalidLanguage models are susceptible to incorrect patient self-diagnosis in medical applications
Large language models (LLMs) are becoming increasingly relevant as a potential tool for healthcare, aiding communication between clinicians, researchers, and patients. However, traditional evaluations of LLMs on medical …
DiagnosticMultiple-choiceCascaded Self-supervised Learning for Subject-independent EEG-based Emotion Recognition
EEG-based Emotion recognition holds significant promise for applications in human-computer interaction, medicine, and neuroscience. While deep learning has shown potential in this field, current approaches usually rely o…
Contrastive LearningDeep LearningEEGEEG Emotion Recognition+2Factors that affect Camera based Self-Monitoring of Vitals in the Wild
The reliability of the results of self monitoring of the vitals in the wild using medical devices or wearables or camera based smart phone solutions is subject to variabilities such as position of placement, hardware of …
PositionSMART: Self-learning Meta-strategy Agent for Reasoning Tasks
Tasks requiring deductive reasoning, especially those involving multiple steps, often demand adaptive strategies such as intermediate generation of rationales or programs, as no single approach is universally optimal. Wh…
GSM8KSelf-Learning