PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
Recently, long chain of thought (LCoT), Large Language Models (LLMs), have taken the machine learning world by storm with their breathtaking reasoning capabilities. However, are the abstract reasoning abilities of these models general enough for problems of practical importance? Unlike past work, which has focused mainly on math, coding, and data wrangling, we focus on a historical linguistics-inspired inductive reasoning problem, formulated as Programming by Examples. We develop a fully automated pipeline for dynamically generating a benchmark for this task with controllable difficulty in order to tackle scalability and contamination issues to which many reasoning benchmarks are subject. Using our pipeline, we generate a test set with nearly 1k instances that is challenging for all state-of-the-art reasoning LLMs, with the best model (Claude-3.7-Sonnet) achieving a mere 54% pass rate, demonstrating that LCoT LLMs still struggle with a class or reasoning that is ubiquitous in historical linguistics as well as many other domains.
Code (0)
등록된 구현이 없습니다.
Tasks
MathMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
LLMs can solve program synthesis tasks but remain inefficient and unreliable on hard instances requiring large combinatorial search. Given a small set of reasoning traces, we use coding agents to compile them into reusab…
Program SynthesisEvaluating Step-by-Step Reasoning through Symbolic Verification
Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be mo…
In-Context LearningPrompt Selection and Augmentation for Few Examples Code Generation in Large Language Model and its Application in Robotics Control
Few-shot prompting and step-by-step reasoning have enhanced the capabilities of Large Language Models (LLMs) in tackling complex tasks including code generation. In this paper, we introduce a prompt selection and augment…
Code GenerationDiversityGSM8KLanguage Modeling+3ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
Though reasoning-based large language models (LLMs) have excelled in mathematics and programming, their capabilities in knowledge-intensive medical question answering remain underexplored. To address this, we introduce R…
Medical Question AnsweringQuestion AnsweringReasoning in Non-Probabilistic Uncertainty: Logic Programming and Neural-Symbolic Computing as Examples
This article aims to achieve two goals: to show that probability is not the only way of dealing with uncertainty (and even more, that there are kinds of uncertainty which are for principled reasons not addressable with p…