Safe Policy Learning for Continuous Control

2019-09-25 · Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, Mohammad Ghavamzadeh

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that keep the agent in desirable situations, both during training and at convergence. We formulate these problems as {\em constrained} Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the selected action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world robot obstacle-avoidance problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlMuJoCo

Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

2022-09-19 · Soutrik Bandyopadhyay, Shubhendu Bhasin

This paper proposes an on-policy reinforcement learning (RL) control algorithm that solves the optimal regulation problem for a class of uncertain continuous-time nonlinear systems under user-defined state constraints. W…

reinforcement-learning Reinforcement Learning (RL)Safe Reinforcement Learning

Safe Reinforcement Learning with Chance-constrained Model Predictive Control

2021-12-27 · Samuel Pfrommer, Tanmay Gautam, Alec Zhou, Somayeh Sojoudi

Real-world reinforcement learning (RL) problems often demand that agents behave safely by obeying a set of designed constraints. We address the challenge of safe RL by coupling a safety guide based on model predictive co…

Model Predictive Control reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Policy Bifurcation in Safe Reinforcement Learning

2024-03-19 · Wenjun Zou, Yao Lyu, Jie Li, Yujie Yang 외

Safe reinforcement learning (RL) offers advanced solutions to constrained optimal control problems. Existing studies in safe RL implicitly assume continuity in policy functions, where policies map states to actions in a …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces

2023-10-15 · Ammar N. Abbas, Georgios C. Chasparis, John D. Kelleher

Traditional controllers have limitations as they rely on prior knowledge about the physics of the problem, require modeling of dynamics, and struggle to adapt to abnormal situations. Deep reinforcement learning has the p…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

Lyapunov-Aware Quantum-Inspired Reinforcement Learning for Continuous-Time Vehicle Control: A Feasibility Study

2025-10-21 · Nutkritta Kraipatthanapong, Natthaphat Thathong, Pannita Suksawas, Thanunnut Klunklin 외 arxiv

This paper presents a novel Lyapunov-Based Quantum Reinforcement Learning (LQRL) framework that integrates quantum policy optimization with Lyapunov stability analysis for continuous-time vehicle control. The proposed ap…

Reinforcement Learning