Policy Optimization Reinforcement Learning with Entropy Regularization
Entropy regularization is an important idea in reinforcement learning, with great success in recent algorithms like Soft Q Network (SQN) and Soft Actor-Critic (SAC1). In this work, we extend this idea into the on-policy realm. We propose the soft policy gradient theorem (SPGT) for on-policy maximum entropy reinforcement learning. With SPGT, a series of new policy optimization algorithms are derived, such as SPG, SA2C, SA3C, SDDPG, STRPO, SPPO, SIMPALA and so on. We find that SDDPG is equivalent to SAC1. For policy gradient, the policy network is often represented as a Gaussian distribution with a global action variance, which damages the representation capacity. We introduce a local action variance for policy network and find it can work collaboratively with the idea of entropy regularization. Our method outperforms prior works on a range of benchmark tasks. Furthermore, our method can be easily extended to large scale experiment with great stability and parallelism.
Code (0)
등록된 구현이 없습니다.
Tasks
Continuous Controlreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalized Maximum Entropy Reinforcement Learning via Reward Shaping
Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy reg…
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization
Trust Region Policy Optimization (TRPO) is a popular and empirically successful policy search algorithm in reinforcement learning (RL). It iteratively solved the surrogate problem which restricts consecutive policies to …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Games
In this paper, we investigate the impact of introducing relative entropy regularization on the Nash Equilibria (NE) of General-Sum $N$-agent games, revealing the fact that the NE of such games conform to linear Gaussian …
Reinforcement Learning (RL)A unified view of entropy-regularized Markov decision processes
We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimizati…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Marginalized State Distribution Entropy Regularization in Policy Optimization
Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stoch…
continuous-controlContinuous ControlReinforcement Learning