Measuring few-shot extrapolation with program induction
Neural networks are capable of learning complex functions, but still have problems generalizing from few examples and beyond their training distribution. Meta-learning provides a paradigm to train networks to learn from few examples, but it has been shown that its most popular benchmarks require very limited generalization capabilities. Program induction lies at the opposite end of the spectrum: programs are capable of extrapolating from very few examples, but we still do not know how to efficiently search for complex programs. We propose a common benchmark for both communities, measuring extrapolation from few examples coming from the execution of small programs. These are obtained by leveraging a C++ interpreter on codes from programming competitions; extracting small sub-codes with their corresponding input-output pairs. Statistical analysis and preliminary human experiments show the potential of this benchmark for enabling progress in few-shot extrapolation.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningProgram inductionSimilar Papers 제목 키워드 기반
Analysis of Error Sources in LLM-based Hypothesis Search for Few-Shot Rule Induction
Inductive reasoning enables humans to infer abstract rules from limited examples and apply them to novel situations. In this work, we compare an LLM-based hypothesis search framework with direct program generation approa…
Neural Program Meta-Induction
Most recently proposed methods for Neural Program Induction work under the assumption of having a large set of input/output (I/O) examples for learning any underlying input-output mapping. This paper aims to address the …
Program inductionTransfer LearningLifting Traces to Logic: Programmatic Skill Induction with Neuro-Symbolic Learning for Long-Horizon Agentic Tasks
Foundation model-driven agents often struggle with long-horizon planning due to the transient nature of purely prompting-based reasoning. While existing skill induction methods mitigate this by distilling experience into…
Compositional diversity in visual concept learning
Humans leverage compositionality to efficiently learn new concepts, understanding how familiar parts can combine together to form novel objects. In contrast, popular computer vision models struggle to make the same types…
DiversityProgram inductionGuided Sketch-Based Program Induction by Search Gradients
Many tasks can be easily solved using machine learning techniques. However, some tasks cannot readily be solved using statistical models, requiring a symbolic approach instead. Program induction is one of the ways that s…
Program induction