Planning with Sequence Models through Iterative Energy Minimization
Recent works have shown that sequence modeling can be effectively used to train reinforcement learning (RL) policies. However, the success of applying existing sequence models to planning, in which we wish to obtain a trajectory of actions to reach some goal, is less straightforward. The typical autoregressive generation procedures of sequence models preclude sequential refinement of earlier steps, which limits the effectiveness of a predicted plan. In this paper, we suggest an approach towards integrating planning with sequence models based on the idea of iterative energy minimization, and illustrate how such a procedure leads to improved RL performance across different tasks. We train a masked language model to capture an implicit energy function over trajectories of actions, and formulate planning as finding a trajectory of actions with minimum energy. We illustrate how this procedure enables improved performance over recent approaches across BabyAI and Atari environments. We further demonstrate unique benefits of our iterative optimization procedure, involving new task generalization, test-time constraints adaptation, and the ability to compose plans together. Project website: https://hychen-naza.github.io/projects/LEAP
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Learning Iterative Reasoning through Energy Minimization
Deep learning has excelled on complex pattern recognition tasks such as image classification and object recognition. However, it struggles with tasks requiring nontrivial reasoning, such as algorithmic computation. Human…
image-classificationImage ClassificationObject RecognitionEnergy Efficient UAV Communication with Energy Harvesting
This paper investigates an unmanned aerial vehicle (UAV)-enabled wireless communication system with energy harvesting, where the UAV transfers energy to the users in half duplex or full duplex, and the users harvest ener…
Learning Iterative Reasoning through Energy Diffusion
We introduce iterative reasoning through energy diffusion (IRED), a novel framework for learning to reason for a variety of tasks by formulating reasoning and decision-making problems with energy-based optimization. IRED…
Decision MakingMatrix CompletionPHASED: Phase-Aware Submodularity-Based Energy Disaggregation
Energy disaggregation is the task of discerning the energy consumption of individual appliances from aggregated measurements, which holds promise for understanding and reducing energy usage. In this paper, we propose PHA…
Convex Optimization for Parallel Energy Minimization
Energy minimization has been an intensely studied core problem in computer vision. With growing image sizes (2D and 3D), it is now highly desirable to run energy minimization algorithms in parallel. But many existing alg…
Denoising