Double Meta-Learning for Data Efficient Policy Optimization in Non-Stationary Environments

2020-11-21 · Elahe Aghapour, Nora Ayanian

We are interested in learning models of non-stationary environments, which can be framed as a multi-task learning problem. Model-free reinforcement learning algorithms can achieve good asymptotic performance in multi-task learning at a cost of extensive sampling, due to their approach, which requires learning from scratch. While model-based approaches are among the most data efficient learning algorithms, they still struggle with complex tasks and model uncertainties. Meta-reinforcement learning addresses the efficiency and generalization challenges on multi task learning by quickly leveraging the meta-prior policy for a new task. In this paper, we propose a meta-reinforcement learning approach to learn the dynamic model of a non-stationary environment to be used for meta-policy optimization later. Due to the sample efficiency of model-based learning methods, we are able to simultaneously train both the meta-model of the non-stationary environment and the meta-policy until dynamic model convergence. Then, the meta-learned dynamic model of the environment will generate simulated data for meta-policy optimization. Our experiment demonstrates that our proposed method can meta-learn the policy in a non-stationary environment with the data efficiency of model-based learning approaches while achieving the high asymptotic performance of model-free meta-reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2011.10714

Code (0)

등록된 구현이 없습니다.

Tasks

Meta-LearningMeta Reinforcement LearningMulti-Task Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On First-Order Meta-Reinforcement Learning with Moreau Envelopes

2023-05-20 · Mohammad Taha Toghani, Sebastian Perez-Salazar, César A. Uribe

Meta-Reinforcement Learning (MRL) is a promising framework for training agents that can quickly adapt to new environments and tasks. In this work, we study the MRL problem under the policy gradient formulation, where we …

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning

Statistically Efficient Off-Policy Policy Gradients

2020-02-10 · ICML 2020 1 · Nathan Kallus, Masatoshi Uehara

Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of p…

Policy Gradient Methods Reinforcement Learning

Near-Optimal Offline Reinforcement Learning via Double Variance Reduction

2021-02-02 · NeurIPS 2021 12 · Ming Yin, Yu Bai, Yu-Xiang Wang

We consider the problem of offline reinforcement learning (RL) -- a well-motivated setting of RL that aims at policy optimization using only historical data. Despite its wide applicability, theoretical understandings of …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Policy Optimization with Variance Regularization

2021-01-01 · Riashat Islam, Samarth Sinha, Homanga Bharadhwaj, Samin Yeasar Arnob 외

Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributiona…

continuous-control Continuous Control Offline RL Reinforcement Learning (RL)

Offline Policy Optimization in RL with Variance Regularizaton

2022-12-29 · Riashat Islam, Samarth Sinha, Homanga Bharadhwaj, Samin Yeasar Arnob 외