MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
Meta reinforcement learning (meta-RL) extracts knowledge from previous tasks and achieves fast adaptation to new tasks. Despite recent progress, efficient exploration in meta-RL remains a key challenge in sparse-reward tasks, as it requires quickly finding informative task-relevant experiences in both meta-training and adaptation. To address this challenge, we explicitly model an exploration policy learning problem for meta-RL, which is separated from exploitation policy learning, and introduce a novel empowerment-driven exploration objective, which aims to maximize information gain for task identification. We derive a corresponding intrinsic reward and develop a new off-policy meta-RL framework, which efficiently learns separate context-aware exploration and exploitation policies by sharing the knowledge of task inference. Experimental evaluation shows that our meta-RL method significantly outperforms state-of-the-art baselines on various sparse-reward MuJoCo locomotion tasks and more complex sparse-reward Meta-World tasks.
Code (1)
Tasks
Efficient ExplorationMeta Reinforcement LearningMuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Empowerment-driven Exploration using Mutual Information Estimation
Exploration is a difficult challenge in reinforcement learning and is of prime importance in sparse reward environments. However, many of the state of the art deep reinforcement learning algorithms, that rely on epsilon-…
Deep Reinforcement LearningMontezuma's RevengeMutual Information Estimationreinforcement-learning+2Towards Empowerment Gain through Causal Structure Learning in Model-Based RL
In Model-Based Reinforcement Learning (MBRL), incorporating causal structures into dynamics models provides agents with a structured understanding of the environments, enabling efficient decision. Empowerment as an intri…
Causal DiscoveryModel-based Reinforcement LearningFlexible Empowerment at Reasoning with Extended Best-of-N Sampling
This paper proposes a novel method that incorporates empowerment when reasoning actions in reinforcement learning (RL), thereby achieving the flexibility of exploration-exploitation dilemma (EED). In previous methods, em…
Reinforcement LearningConstrained Ensemble Exploration for Unsupervised Skill Discovery
Unsupervised Reinforcement Learning (RL) provides a promising paradigm for learning useful behaviors via reward-free per-training. Existing methods for unsupervised RL mainly conduct empowerment-driven skill discovery or…
Reinforcement Learning (RL)Unsupervised Reinforcement LearningUnsupervised Skill Discovery through Skill Regions Differentiation
Unsupervised Reinforcement Learning (RL) aims to discover diverse behaviors that can accelerate the learning of downstream tasks. Previous methods typically focus on entropy-based exploration or empowerment-driven skill …
Density EstimationReinforcement Learning (RL)Unsupervised Reinforcement Learning