Hindsight Trust Region Policy Optimization

2019-07-29 · Hanbo Zhang, Site Bai, Xuguang Lan, David Hsu, Nanning Zheng

Reinforcement Learning(RL) with sparse rewards is a major challenge. We propose \emph{Hindsight Trust Region Policy Optimization}(HTRPO), a new RL algorithm that extends the highly successful TRPO algorithm with \emph{hindsight} to tackle the challenge of sparse rewards. Hindsight refers to the algorithm's ability to learn from information across goals, including ones not intended for the current task. HTRPO leverages two main ideas. It introduces QKL, a quadratic approximation to the KL divergence constraint on the trust region, leading to reduced variance in KL divergence estimation and improved stability in policy update. It also presents Hindsight Goal Filtering(HGF) to select conductive hindsight goals. In experiments, we evaluate HTRPO in various sparse reward tasks, including simple benchmarks, image-based Atari games, and simulated robot control. Ablation studies indicate that QKL and HGF contribute greatly to learning stability and high performance. Comparison results show that in all tasks, HTRPO consistently outperforms both TRPO and HPG, a state-of-the-art algorithm for RL with sparse rewards.

📄 PDF Abstract BibTeX arXiv:1907.12439

Code (1)

HTRPOCODES/HTRPO-v2 공식 구현 pytorch

Tasks

Atari GamesPolicy Gradient MethodsReinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Data-efficient Hindsight Off-policy Option Learning

2020-07-30 · Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe 외

We introduce Hindsight Off-policy Options (HO2), a data-efficient option learning algorithm. Given any trajectory, HO2 infers likely option choices and backpropagates through the dynamic programming inference procedure t…

Robot Manipulation

Trust-Region-Free Policy Optimization for Stochastic Policies

2023-02-15 · Mingfei Sun, Benjamin Ellis, Anuj Mahajan, Sam Devlin 외

Trust Region Policy Optimization (TRPO) is an iterative method that simultaneously maximizes a surrogate objective and enforces a trust region constraint over consecutive policies in each iteration. The combination of th…

Learning to Constrain Policy Optimization with Virtual Trust Region

2022-04-20 · Hung Le, Thommen Karimpanal George, Majid Abdolshah, Dung Nguyen 외

We introduce a constrained optimization method for policy gradient reinforcement learning, which uses a virtual trust region to regulate each policy update. In addition to using the proximity of one single old policy as …

Atari Games Policy Gradient Methods

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

2017-07-06 · ICLR 2018 1 · Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically re…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

2026-02-04 · Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim 외 arxiv

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as globa…

Reinforcement Learning Mathematical Reasoning