Lagrangian-based online safe reinforcement learning for state-constrained systems

2023-05-22 · Soutrik Bandyopadhyay, Shubhendu Bhasin

This paper proposes a safe reinforcement learning (RL) algorithm that approximately solves the state-constrained optimal control problem for continuous-time uncertain nonlinear systems. We formulate the safe RL problem as the minimization of a Lagrangian that includes the cost functional and a user-defined barrier Lyapunov function (BLF) encoding the state constraints. We show that the analytical solution obtained by the application of Karush-Kuhn-Tucker (KKT) conditions contains a state-dependent expression for the Lagrange multiplier, which is a function of uncertain terms in the system dynamics. We argue that a naive estimation of the Lagrange multiplier may lead to safety constraint violations. To obviate this challenge, we propose an Actor-Critic-Identifier-Lagrangian (ACIL) algorithm that learns optimal control policies from online data without compromising safety. We provide safety and boundedness guarantees with the proposed algorithm and compare its performance with existing offline/online RL methods via a simulation study.

📄 PDF Abstract BibTeX arXiv:2305.12967

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning

2023-05-31 · Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang 외

We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on expected total utilities. Our focus is con…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

2023-02-25 · Rahul Misra, Rafał Wisniewski, Carsten Skovmose Kallesøe

We study optimality for the safety-constrained Markov decision process which is the underlying framework for safe reinforcement learning. Specifically, we consider a constrained Markov decision process (with finite state…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Adaptive Primal-Dual Method for Safe Reinforcement Learning

2024-02-01 · Weiqin Chen, James Onyejizu, Long Vu, Lan Hoang 외

Primal-dual methods have a natural application in Safe Reinforcement Learning (SRL), posed as a constrained policy optimization problem. In practice however, applying primal-dual methods to SRL is challenging, due to the…

reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

2021-02-17 · Baiyu Peng, Yao Mu, Jingliang Duan, Yang Guan 외

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to repr…

Autonomous Driving reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

2021-08-26 · Baiyu Peng, Jingliang Duan, Jianyu Chen, Shengbo Eben Li 외

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constraine…

reinforcement-learning Reinforcement Learning (RL)Robot Navigation