EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization

2021-10-26 · Sahar Roostaie, Mohammad Mehdi Ebadzadeh

Trust Region Policy Optimization (TRPO) is a popular and empirically successful policy search algorithm in reinforcement learning (RL). It iteratively solved the surrogate problem which restricts consecutive policies to be close to each other. TRPO is an on-policy algorithm. On-policy methods bring many benefits, like the ability to gauge each resulting policy. However, they typically discard all the knowledge about the policies which existed before. In this work, we use a replay buffer to borrow from the off-policy learning setting to TRPO. Entropy regularization is usually used to improve policy optimization in reinforcement learning. It is thought to aid exploration and generalization by encouraging more random policy choices. We add an Entropy regularization term to advantage over {\pi}, accumulated over time steps, in TRPO. We call this update EnTRPO. Our experiments demonstrate EnTRPO achieves better performance for controlling a Cart-Pole system compared with the original TRPO

📄 PDF Abstract BibTeX arXiv:2110.13373

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Entropy Regularization 설명 없음

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

2026-02-04 · Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim 외 arxiv

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as globa…

Reinforcement Learning Mathematical Reasoning

Compatible Natural Gradient Policy Search

2019-02-07 · Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters 외

Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural …

continuous-control Continuous Control

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

2017-07-06 · ICLR 2018 1 · Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically re…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

TEAC: Intergrating Trust Region and Max Entropy Actor Critic for Continuous Control

2021-01-01 · Hongyu Zang, Xin Li, Li Zhang, Peiyao Zhao 외

Trust region methods and maximum entropy methods are two state-of-the-art branches used in reinforcement learning (RL) for the benefits of stability and exploration in continuous environments, respectively. This paper pr…

continuous-control Continuous Control MuJoCo Reinforcement Learning (RL)

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization

2020-07-13 · Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei 외

Natural policy gradient (NPG) methods are among the most widely used policy optimization algorithms in contemporary reinforcement learning. This class of methods is often applied in conjunction with entropy regularizatio…

Policy Gradient Methods