Large Language Models' Reasoning Stalls: An Investigation into the Capabilities of Frontier Models
Empirical methods to examine the capability of Large Language Models (LLMs) to use Automated Theorem Prover (ATP) reasoning strategies are studied. We evaluate the performance of State of the Art models from December 2023 and August 2024 on PRONTOQA steamroller reasoning problems. For that, we develop methods for assessing LLM response accuracy and correct answer correlation. Our results show that progress in improving LLM reasoning abilities has stalled over the nine month period. By tracking completion tokens, we show that almost all improvement in reasoning ability since GPT-4 was released can be attributed to either hidden system prompts or the training of models to automatically use generic Chain of Thought prompting strategies. Among the ATP reasoning strategies tried, we found that current frontier LLMs are best able to follow the bottom-up (also known as forward-chaining) strategy. A low positive correlation was found between an LLM response containing correct reasoning and arriving at the correct conclusion.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Complementarity Between Paid and Organic Installs in Mobile App Advertising
Prior spending shutoff experiments in search advertising have found that paid ads cannibalize organic traffic. But it is unclear whether the same is true for other high volume advertising channels like mobile display adv…
MarketingDissertation on Applied Microeconomics of Freemium Pricing Strategies in Mobile App Market
In my dissertation, I will analyze how the product market position of a mobile app affects its pricing strategies, which in turn impacts an app's monetization process. Using natural language processing and k-mean cluster…
Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs
We introduce Cube Bench, a Rubik's-cube benchmark for evaluating spatial and sequential reasoning in multimodal large language models (MLLMs). The benchmark decomposes performance into five skills: (i) reconstructing cub…
Spatial ReasoningVisual ReasoningBeyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
Enhancing the complex reasoning capabilities of Large Language Models (LLMs) attracts widespread attention. While reinforcement learning (RL) has shown superior performance for improving complex reasoning, its impact on …
Reinforcement LearningBase Models Know How to Reason, Thinking Models Learn When
What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training small Sparse Autoencoders on sentence-l…