Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

2021-02-17 · Baiyu Peng, Yao Mu, Jingliang Duan, Yang Guan, Shengbo Eben Li, Jianyu Chen

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to represent the requirements in real-world environment with uncertainty. Existing chance constrained RL methods like the penalty method and the Lagrangian method either exhibit periodic oscillations or cannot satisfy the constraints. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively. Then, a proportional-integral Lagrangian method is proposed to steady learning process while improving safety. To prevent integral overshooting and reduce conservatism, we introduce the integral separation technique inspired by PID control. Finally, an analytical gradient of the chance constraint is utilized for model-based policy optimization. The effectiveness of SPIL is demonstrated by a narrow car-following task. Experiments indicate that compared with previous methods, SPIL improves the performance while guaranteeing safety, with a steady learning process.

📄 PDF Abstract BibTeX arXiv:2102.08539

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Driving reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

2021-08-26 · Baiyu Peng, Jingliang Duan, Jianyu Chen, Shengbo Eben Li 외

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constraine…

reinforcement-learning Reinforcement Learning (RL)Robot Navigation

PID Optimization Using Lagrangian Mechanics

2023-09-25 · Ethan Kou, Majid Moghadam

Creating a simulation of a system enables the tuning of control systems without the need for a physical system. In this paper, we employ Lagrangian Mechanics to derive a set of equations to simulate an inverted pendulum …

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

2020-07-08 · Adam Stooke, Joshua Achiam, Pieter Abbeel

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Predictive Lagrangian Optimization for Constrained Reinforcement Learning

2025-01-25 · Tianqi Zhang, Puzhen Yuan, Guojian Zhan, Ziyu Lin 외

Constrained optimization is popularly seen in reinforcement learning for addressing complex control tasks. From the perspective of dynamic system, iteratively solving a constrained optimization problem can be framed as t…

Model Predictive Control reinforcement-learning Reinforcement Learning

Job Scheduling in Datacenters using Constraint Controlled RL

2022-11-10 · Vanamala Venkataswamy

This paper studies a model for online job scheduling in green datacenters. In green datacenters, resource availability depends on the power supply from the renewables. Intermittent power supply from renewables leads to i…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)Scheduling