Constrained Variational Policy Optimization for Safe Reinforcement Learning

2022-01-28 · Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Zhiwei Steven Wu, Bo Li, Ding Zhao

Safe reinforcement learning (RL) aims to learn policies that satisfy certain constraints before deploying them to safety-critical applications. Previous primal-dual style approaches suffer from instability issues and lack optimality guarantees. This paper overcomes the issues from the perspective of probabilistic inference. We introduce a novel Expectation-Maximization approach to naturally incorporate constraints during the policy learning: 1) a provable optimal non-parametric variational distribution could be computed in closed form after a convex optimization (E-step); 2) the policy parameter is improved within the trust region based on the optimal variational distribution (M-step). The proposed algorithm decomposes the safe RL problem into a convex optimization phase and a supervised learning phase, which yields a more stable training performance. A wide range of experiments on continuous robotic tasks shows that the proposed method achieves significantly better constraint satisfaction performance and better sample efficiency than baselines. The code is available at https://github.com/liuzuxin/cvpo-safe-rl.

📄 PDF Abstract BibTeX arXiv:2201.11927

Code (2)

liuzuxin/cvpo-safe-rl 공식 구현 pytorch

zifanwu/cal pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning

2023-10-05 · NeurIPS 2023 11 · Yihang Yao, Zuxin Liu, Zhepeng Cen, Jiacheng Zhu 외

Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements duri…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning+1

Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning

2024-12-11 · Prajwal Koirala, Zhanhong Jiang, Soumik Sarkar, Cody Fleming

In safe offline reinforcement learning (RL), the objective is to develop a policy that maximizes cumulative rewards while strictly adhering to safety constraints, utilizing only offline data. Traditional methods often fa…

Autonomous Driving Offline RL reinforcement-learning Reinforcement Learning+2

Variational Constrained Reinforcement Learning with Application to Planning at Roundabout

2019-09-25 · Yuan Tian, Minghao Han, Lixian Zhang, Wulong Liu 외

Planning at roundabout is crucial for autonomous driving in urban and rural environments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into accou…

Autonomous Driving reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Convergent Policy Optimization for Safe Reinforcement Learning

2019-10-26 · NeurIPS 2019 12 · Ming Yu, Zhuoran Yang, Mladen Kolar, Zhaoran Wang

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonco…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Embedding Safety into RL: A New Take on Trust Region Methods

2024-11-05 · Nikola Milosevic, Johannes Müller, Nico Scherf

Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrif…

Reinforcement Learning (RL)