Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
The goal of meta-reinforcement learning (meta-RL) is to build agents that can quickly learn new tasks by leveraging prior experience on related tasks. Learning a new task often requires both exploring to gather task-relevant information and exploiting this information to solve the task. In principle, optimal exploration and exploitation can be learned end-to-end by simply maximizing task performance. However, such meta-RL approaches struggle with local optima due to a chicken-and-egg problem: learning to explore requires good exploitation to gauge the exploration's utility, but learning to exploit requires information gathered via exploration. Optimizing separate objectives for exploration and exploitation can avoid this problem, but prior meta-RL exploration objectives yield suboptimal policies that gather information irrelevant to the task. We alleviate both concerns by constructing an exploitation objective that automatically identifies task-relevant information and an exploration objective to recover only this information. This avoids local optima in end-to-end training, without sacrificing optimal exploration. Empirically, DREAM substantially outperforms existing approaches on complex meta-RL problems, such as sparse-reward 3D visual navigation. Videos of DREAM: https://ezliu.github.io/dream/
Code (2)
Tasks
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Visual NavigationSimilar Papers 제목 키워드 기반
Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
Despite the close connection between exploration and sample efficiency, most state of the art reinforcement learning algorithms include no considerations for exploration beyond maximizing the entropy of the policy. In th…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward t…
Efficient ExplorationMeta Reinforcement LearningMuJoCoreinforcement-learning+2MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
Large Language Model (LLM) agents have demonstrated remarkable proficiency in learned tasks, yet they often struggle to adapt to non-stationary environments with feedback. While In-Context Learning and external memory of…
Reinforcement LearningEfficient Reinforcement Learning via Decoupling Exploration and Utilization
Reinforcement Learning (RL), recognized as an efficient learning approach, has achieved remarkable success across multiple fields and applications, including gaming, robotics, and autonomous vehicles. Classical single-ag…
Autonomous VehiclesMuJoCoreinforcement-learningReinforcement Learning+1Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration
Intrinsic rewards can improve exploration in reinforcement learning, but the exploration process may suffer from instability caused by non-stationary reward shaping and strong dependency on hyperparameters. In this work,…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)