Learning to Explore via Meta-Policy Gradient

2018-07-01 · ICML 2018 7 · Tianbing Xu, Qiang Liu, Liang Zhao, Jian Peng

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning continuous control tasks.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Weight Decay 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Adam 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Batch Normalization 설명 없음

DDPG DDPG, or Deep Deterministic Policy Gradient, is an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action…

Learning to Explore with Meta-Policy Gradient

2018-03-13 · Tianbing Xu, Qiang Liu, Liang Zhao, Jian Peng

Q-Learning Reinforcement Learning Reinforcement Learning (RL)

Interpretable Meta-Reinforcement Learning with Actor-Critic Method

2021-01-01 · Xingyuan Liang, Xu-Ying Liu

Meta-reinforcement learning (meta-RL) algorithms have successfully trained agent systems to perform well on different tasks within only few updates. However, in gradient-based meta-RL algorithms, the Q-function at adapta…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ProMP: Proximal Meta-Policy Search

2018-10-16 · ICLR 2019 5 · Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour 외

Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-effic…

Meta-Learning Meta Reinforcement Learning Reinforcement Learning

On First-Order Meta-Reinforcement Learning with Moreau Envelopes

2023-05-20 · Mohammad Taha Toghani, Sebastian Perez-Salazar, César A. Uribe

Meta-Reinforcement Learning (MRL) is a promising framework for training agents that can quickly adapt to new environments and tasks. In this work, we study the MRL problem under the policy gradient formulation, where we …

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning

On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies

2022-09-21 · Haozhi Wang, Qing Wang, Yunfeng Shao, Dong Li 외

Modern meta-reinforcement learning (Meta-RL) methods are mainly developed based on model-agnostic meta-learning, which performs policy gradient steps across tasks to maximize policy performance. However, the gradient con…

continuous-control Continuous Control Meta-Learning Meta Reinforcement Learning+3