Optimization Issues in KL-Constrained Approximate Policy Iteration

2021-02-11 · Nevena Lazić, Botao Hao, Yasin Abbasi-Yadkori, Dale Schuurmans, Csaba Szepesvári

Many reinforcement learning algorithms can be seen as versions of approximate policy iteration (API). While standard API often performs poorly, it has been shown that learning can be stabilized by regularizing each policy update by the KL-divergence to the previous policy. Popular practical algorithms such as TRPO, MPO, and VMPO replace regularization by a constraint on KL-divergence of consecutive policies, arguing that this is easier to implement and tune. In this work, we study this implementation choice in more detail. We compare the use of KL divergence as a constraint vs. as a regularizer, and point out several optimization issues with the widely-used constrained approach. We show that the constrained algorithm is not guaranteed to converge even on simple problem instances where the constrained problem can be solved exactly, and in fact incurs linear expected regret. With approximate implementation using softmax policies, we show that regularization can improve the optimization landscape of the original objective. We demonstrate these issues empirically on several bandit and RL environments.

📄 PDF Abstract BibTeX arXiv:2102.06234

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints

2019-11-26 · Jingliang Duan, Zhengyu Liu, Shengbo Eben Li, Qi Sun 외

This paper presents a constrained adaptive dynamic programming (CADP) algorithm to solve general nonlinear nonaffine optimal control problems with known dynamics. Unlike previous ADP algorithms, it can directly deal with…

Proactive Constrained Policy Optimization with Preemptive Penalty

2025-08-03 · Ning Yang, Pengyu Wang, Guoqing Liu, Haifeng Zhang 외 arxiv

Safe Reinforcement Learning (RL) often faces significant issues such as constraint violations and instability, necessitating the use of constrained policy optimization, which seeks optimal policies while ensuring adheren…

Reinforcement Learning

Penalized Proximal Policy Optimization for Safe Reinforcement Learning

2022-05-24 · Linrui Zhang, Li Shen, Long Yang, Shixiang Chen 외

Safe reinforcement learning aims to learn the optimal policy while satisfying safety constraints, which is essential in real-world applications. However, current algorithms still struggle for efficient policy updates wit…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality

2025-08-24 · Shaocong Ma, Ziyi Chen, Yi Zhou, Heng Huang arxiv

The goal of robust constrained reinforcement learning (RL) is to optimize an agent's performance under the worst-case model uncertainty while satisfying safety or resource constraints. In this paper, we demonstrate that …

Reinforcement Learning

On Connections between Constrained Optimization and Reinforcement Learning

2019-10-18 · Nino Vieillard, Olivier Pietquin, Matthieu Geist

Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this paper, we draw connections between DP and…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)