Off-Policy Primal-Dual Safe Reinforcement Learning

2024-01-26 · Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, Dong Wang

Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

📄 PDF Abstract BibTeX arXiv:2401.14758

Code (2)

pku-alignment/omnisafe 공식 구현 pytorch

zifanwu/cal 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement LearningSafe Reinforcement Learning

Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

2018-02-19 · Qingkai Liang, Fanyu Que, Eytan Modiano

Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

2026-06-12 · Ayoub Belouadah, Sylvain Kubler, Yves Le Traon arxiv

Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep…

Reinforcement Learning

Adaptive Primal-Dual Method for Safe Reinforcement Learning

2024-02-01 · Weiqin Chen, James Onyejizu, Long Vu, Lan Hoang 외

Primal-dual methods have a natural application in Safe Reinforcement Learning (SRL), posed as a constrained policy optimization problem. In practice however, applying primal-dual methods to SRL is challenging, due to the…

reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee

2020-11-11 · Tengyu Xu, Yingbin Liang, Guanghui Lan

In safe reinforcement learning (SRL) problems, an agent explores the environment to maximize an expected total reward and meanwhile avoids violation of certain constraints on a number of expected total costs. In general,…

reinforcement-learning Reinforcement Learning (RL)Safe Reinforcement Learning

Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual

2026-02-25 · Yining Li, Peizhong Ju, Ness Shroff arxiv

Reinforcement Learning from Human Feedback (RLHF) plays a significant role in aligning Large Language Models (LLMs) with human preferences. While RLHF with expected reward constraints can be formulated as a primal-dual o…