Program Guided Agent
Developing agents that can learn to follow natural language instructions has been an emerging research direction. While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans. To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks. We then devise a modular framework that learns to perform a task specified by a program – as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task. Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy. We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.
Code (0)
등록된 구현이 없습니다.
Tasks
MinecraftZero-shot GeneralizationSimilar Papers 제목 키워드 기반
ProTo: Program-Guided Transformer for Program-Guided Tasks
Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision …
Decision MakingLearning to ExecuteMinecraftVisual Question Answering (VQA)+1Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
A key challenge for reinforcement learning is solving long-horizon planning problems. Recent work has leveraged programs to guide reinforcement learning in these settings. However, these approaches impose a high manual b…
MinecraftProgram Synthesisreinforcement-learningReinforcement Learning+1AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management
The rapid development of mobile GUI agents has stimulated growing research interest in long-horizon task automation. However, building agents for these tasks faces a critical bottleneck: the reliance on ever-expanding in…
Neural Guided Constraint Logic Programming for Program Synthesis
Synthesizing programs using example input/outputs is a classic problem in artificial intelligence. We present a method for solving Programming By Example (PBE) problems by using a neural model to guide the search of a co…
Graph Neural NetworkProgram SynthesisHeuristically Guided Compilation for Multi-Agent Path Finding
Multi-agent path finding (MAPF) is a task of finding non-conflicting paths connecting agents' specified initial and goal positions in a shared environment. We focus on compilation-based solvers in which the MAPF problem …
Multi-Agent Path Finding