Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning

2019-09-07 · Wenjie Shi, Shiji Song, Cheng Wu

Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.

📄 PDF Abstract BibTeX arXiv:1909.03198

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Generalized Maximum Entropy Reinforcement Learning via Reward Shaping

2021-09-29 · Feng Tao, Yongcan Cao

Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy reg…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Policy Optimization Reinforcement Learning with Entropy Regularization

2019-12-02 · Jingbin Liu, Xinyang Gu, Shuai Liu

Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy …

Continuous Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

S2VG: Soft Stochastic Value Gradient method

2019-09-25 · Xiaoyu Tan, Chao Qu, Junwu Xiong, James Zhang

Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ea…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Soft Actor-Critic with Cross-Entropy Policy Optimization

2021-12-21 · Zhenyang Shi, Surya P. N. Singh

Soft Actor-Critic (SAC) is one of the state-of-the-art off-policy reinforcement learning (RL) algorithms that is within the maximum entropy based RL framework. SAC is demonstrated to perform very well in a list of contin…

Reinforcement Learning (RL)

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

2018-01-04 · ICML 2018 7 · Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine

A platform for Applied Reinforcement Learning (Applied RL)

Continuous Control Decision Making Deep Reinforcement Learning Omniverse Isaac Gym+5