Model-Based Reinforcement Learning via Meta-Policy Optimization
Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.
Code (1)
Tasks
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Double Meta-Learning for Data Efficient Policy Optimization in Non-Stationary Environments
We are interested in learning models of non-stationary environments, which can be framed as a multi-task learning problem. Model-free reinforcement learning algorithms can achieve good asymptotic performance in multi-tas…
Meta-LearningMeta Reinforcement LearningMulti-Task Learningreinforcement-learning+2On First-Order Meta-Reinforcement Learning with Moreau Envelopes
Meta-Reinforcement Learning (MRL) is a promising framework for training agents that can quickly adapt to new environments and tasks. In this work, we study the MRL problem under the policy gradient formulation, where we …
Meta Reinforcement Learningreinforcement-learningReinforcement LearningMeta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel…
AllBilevel OptimizationMeta Reinforcement Learningreinforcement-learning+2Model-Based Offline Meta-Reinforcement Learning with Regularization
Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising appro…
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Surrogate Learning in Meta-Black-Box Optimization: A Preliminary Study
Recent Meta-Black-Box Optimization (MetaBBO) approaches have shown possibility of enhancing the optimization performance through learning meta-level policies to dynamically configure low-level optimizers. However, existi…
Kolmogorov-Arnold NetworksReinforcement Learning (RL)