Testing GPT-4-o1-preview on math and science problems: A follow-up study
In August 2023, Scott Aaronson and I reported the results of testing GPT4 with the Wolfram Alpha and Code Interpreter plug-ins over a collection of 105 original high-school level and college-level science and math problems (Davis and Aaronson, 2023). In September 2024, I tested the recently released model GPT-4o1-preview on the same collection. Overall I found that performance had significantly improved, but was still considerably short of perfect. In particular, problems that involve spatial reasoning are often stumbling blocks.
Code (0)
등록된 구현이 없습니다.
Tasks
MathSpatial ReasoningSimilar Papers 제목 키워드 기반
Evaluation of OpenAI o1: Opportunities and Challenges of AGI
This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natura…
Emotion RecognitionLarge Language ModelMathematical ReasoningNatural Language Inference+2Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems
Benchmarks are critical for measuring progress of math reasoning abilities of Large Language Models (LLMs). However, existing widely-used benchmarks such as GSM8K have been rendered less useful as multiple cutting-edge L…
GSM8KMathMathematical ReasoningUTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts
The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problem…
Code GenerationGSM8KMathMathematical ReasoningReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates
We present that hierarchical LLM reasoning via scaling thought templates can effectively optimize the reasoning search space and outperform the mathematical reasoning capabilities of powerful LLMs like OpenAI o1-preview …
Hierarchical Reinforcement LearningLanguage ModelingMathMathematical ReasoningEnhancing Math Reasoning in Small-sized LLMs via Preview Difficulty-Aware Intervention
Reinforcement learning scaling enhances the reasoning capabilities of large language models, with reinforcement learning serving as the key technique to draw out complex reasoning. However, key technical details of state…
Reinforcement Learning