Supervised Policy Update for Deep Reinforcement Learning

2018-05-29 · ICLR 2019 · Quan Vuong, Yiming Zhang, Keith W. Ross

We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks.

📄 PDF Abstract BibTeX arXiv:1805.11706

Code (1)

quanvuong/Supervised_Policy_Update 공식 구현 tf

Tasks

Deep Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

SUPERVISED POLICY UPDATE

2019-05-01 · ICLR 2019 5 · Quan Vuong, Yiming Zhang, Keith W. Ross

Deep Reinforcement Learning MuJoCo Reinforcement Learning Reinforcement Learning (RL)

Learning Exploration Policies for Model-Agnostic Meta-Reinforcement Learning

2019-05-16 · ICML Workshop AMTL 2019 6 · Anonymous

Meta-Reinforcement learning approaches aim to develop learning procedures that can adapt quickly to a distribution of tasks with the help of a few examples. Developing efficient exploration strategies capable of finding …

Efficient Exploration Meta Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Guide Actor-Critic for Continuous Control

2017-05-22 · ICLR 2018 1 · Voot Tangkaratt, Abbas Abdolmaleki, Masashi Sugiyama

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Self-Supervised Continuous Control without Policy Gradient

2021-01-01 · Hao Sun, Ziping Xu, Meng Fang, Yuhang Song 외

Despite the remarkable progress made by the policy gradient algorithms in reinforcement learning (RL), sub-optimal policies usually result from the local exploration property of the policy gradient update. In this work, …

continuous-control Continuous Control MuJoCo Policy Gradient Methods+3

Entropy Regularized Reinforcement Learning with Cascading Networks

2022-10-16 · Riccardo Della Vecchia, Alena Shilova, Philippe Preux, Riad Akrour

Deep Reinforcement Learning (Deep RL) has had incredible achievements on high dimensional problems, yet its learning process remains unstable even on the simplest tasks. Deep RL uses neural networks as function approxima…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)