Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs

2019-09-06 · Lior Shani, Yonathan Efroni, Shie Mannor

Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspired by Conservative Policy Iteration (CPI). We show that the adaptive scaling mechanism used in TRPO is in fact the natural "RL version" of traditional trust-region methods from convex analysis. We first analyze TRPO in the planning setting, in which we have access to the model and the entire state space. Then, we consider sample-based TRPO and establish $\tilde O(1/\sqrt{N})$ convergence rate to the global optimum. Importantly, the adaptive scaling mechanism allows us to analyze TRPO in regularized MDPs for which we prove fast rates of $\tilde O(1/N)$, much like results in convex optimization. This is the first result in RL of better rates when regularizing the instantaneous cost or reward.

📄 PDF Abstract BibTeX arXiv:1909.02769

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

2026-02-04 · Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim 외 arxiv

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as globa…

Reinforcement Learning Mathematical Reasoning

Provably Convergent Policy Optimization via Metric-aware Trust Region Methods

2023-06-25 · Jun Song, Niao He, Lijun Ding, Chaoyue Zhao

Trust-region methods based on Kullback-Leibler divergence are pervasively used to stabilize policy optimization in reinforcement learning. In this paper, we exploit more flexible metrics and examine two natural extension…

continuous-control Continuous Control Policy Gradient Methods

Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy

2019-06-25 · Boyi Liu, Qi Cai, Zhuoran Yang, Zhaoran Wang

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy

2019-12-01 · NeurIPS 2019 12 · Boyi Liu, Qi Cai, Zhuoran Yang, Zhaoran Wang

Deep Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

Trust Region-Guided Proximal Policy Optimization

2019-01-29 · NeurIPS 2019 12 · Yuhui Wang, Hao He, Xiaoyang Tan, Yaozhong Gan

Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method,…