Too long; didn't solve
Mathematical benchmarks consisting of a range of mathematics problems are widely used to evaluate the reasoning abilities of large language models, yet little is known about how their structural properties influence model behaviour. In this work, we investigate two structural length variables, prompt length and solution length, and analyse how they relate to model performance on a newly constructed adversarial dataset of expert-authored mathematics problems. We find that both prompt and solution lengths correlate positively with increased model failure across models. We also include a secondary, exploratory analysis of cross-model disagreement. Under a difficulty-adjusted normalised analysis, both variables retain weak negative associations with realised model separation, slightly stronger for prompt length. Overall, our main robust finding is that structural length is linked to empirical difficulty in this dataset.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CDIDN: A Registration Model with High Deformation Impedance Capability for Long-Term Tracking of Pulmonary Lesion Dynamics
We study the problem of registration for medical CT images from a novel perspective -- the sensitivity to degree of deformations in CT images. Although some learning-based methods have shown success in terms of average a…
Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels
Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult. We argue that novels provide a …
Language ModelingLanguage ModellingLong-Context UnderstandingCo-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
In reading comprehension, generating sentence-level distractors is a significant task, which requires a deep understanding of the article and question. The traditional entity-centered methods can only generate word-level…
DecoderDistractor GenerationReading ComprehensionSemantic Similarity+2Modeling High-order Interactions across Multi-interests for Micro-video Recommendation
Personalized recommendation system has become pervasive in various video platform. Many effective methods have been proposed, but most of them didn't capture the user's multi-level interest trait and dependencies between…
Vocal Bursts Intensity PredictionTask Proposal: The TL;DR Challenge
The TL;DR challenge fosters research in abstractive summarization of informal text, the largest and fastest-growing source of textual data on the web, which has been overlooked by summarization research so far. The chall…
Abstractive Text SummarizationInformation RetrievalText GenerationText Summarization