Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge
The IPARC Challenge, inspired by ARC, provides controlled program synthesis tasks over synthetic images to evaluate automatic program construction, focusing on sequence, selection, and iteration. This set of 600 tasks has resisted automated solutions. This paper presents a structured inductive programming approach with LLMs that successfully solves tasks across all IPARC categories. The controlled nature of IPARC reveals insights into LLM-based code generation, including the importance of prior structuring, LLMs' ability to aid structuring (requiring human refinement), the need to freeze correct code, the efficiency of code reuse, and how LLM-generated code can spark human creativity. These findings suggest valuable mechanisms for human-LLM collaboration in tackling complex program synthesis.
Code (0)
등록된 구현이 없습니다.
Tasks
ARCCode GenerationProgram SynthesisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis
Many structured prediction and reasoning tasks can be framed as program synthesis problems, where the goal is to generate a program in a domain-specific language (DSL) that transforms input data into the desired output. …
Program SynthesisStructured PredictionANPL: Towards Natural Programming with Interactive Decomposition
Though LLMs are capable of generating plausible programs, it's challenging to interact with the LLMs further to revise the program, especially if the user's specific requirements are different from the initial proposal. …
ARCCode GenerationHumanEvalProgram SynthesisImproving Neural Program Synthesis with Inferred Execution Traces
The task of program synthesis, or automatically generating programs that are consistent with a provided specification, remains a challenging task in artificial intelligence. As in other fields of AI, deep learning-based …
Program SynthesisGPU accelerated program synthesis: Enumerate semantics, not syntax!
Program synthesis is an umbrella term for generating programs and logical formulae from specifications. With the remarkable performance improvements that GPUs enable for deep learning, a natural question arose: can we al…
CPUGPUProgram SynthesisPoE-World: Compositional World Modeling with Products of Programmatic Experts
Learning how the world works is central to building AI agents that can adapt to complex environments. Traditional world models based on deep learning demand vast amounts of training data, and do not flexibly update their…
Montezuma's RevengeProgram Synthesis