홈 › Papers

Predictive Lagrangian Optimization for Constrained Reinforcement Learning

2025-01-25 · Tianqi Zhang, Puzhen Yuan, Guojian Zhan, Ziyu Lin, Yao Lyu, Zhenzhi Qin, Jingliang Duan, Liping Zhang, Shengbo Eben Li

Constrained optimization is popularly seen in reinforcement learning for addressing complex control tasks. From the perspective of dynamic system, iteratively solving a constrained optimization problem can be framed as the temporal evolution of a feedback control system. Classical constrained optimization methods, such as penalty and Lagrangian approaches, inherently use proportional and integral feedback controllers. In this paper, we propose a more generic equivalence framework to build the connection between constrained optimization and feedback control system, for the purpose of developing more effective constrained RL algorithms. Firstly, we define that each step of the system evolution determines the Lagrange multiplier by solving a multiplier feedback optimal control problem (MFOCP). In this problem, the control input is multiplier, the state is policy parameters, the dynamics is described by policy gradient descent, and the objective is to minimize constraint violations. Then, we introduce a multiplier guided policy learning (MGPL) module to perform policy parameters updating. And we prove that the resulting optimal policy, achieved through alternating MFOCP and MGPL, aligns with the solution of the primal constrained RL problem, thereby establishing our equivalence framework. Furthermore, we point out that the existing PID Lagrangian is merely one special case within our framework that utilizes a PID controller. We also accommodate the integration of other various feedback controllers, thereby facilitating the development of new algorithms. As a representative, we employ model predictive control (MPC) as the feedback controller and consequently propose a new algorithm called predictive Lagrangian optimization (PLO). Numerical experiments demonstrate its superiority over the PID Lagrangian method, achieving a larger feasible region up to 7.2% and a comparable average reward.

📄 PDF Abstract BibTeX arXiv:2501.15217

Code (0)

등록된 구현이 없습니다.

Tasks

Model Predictive Controlreinforcement-learning Reinforcement Learning

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

2020-07-08 · Adam Stooke, Joshua Achiam, Pieter Abbeel

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

2023-06-13 · Kihyuk Hong, Yuhang Li, Ambuj Tewari

Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning

2023-05-31 · Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang 외

We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on expected total utilities. Our focus is con…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Safe Reinforcement Learning

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

2023-02-25 · Rahul Misra, Rafał Wisniewski, Carsten Skovmose Kallesøe

We study optimality for the safety-constrained Markov decision process which is the underlying framework for safe reinforcement learning. Specifically, we consider a constrained Markov decision process (with finite state…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning

2021-05-20 · Yongfeng Li, Mingming Zhao, WeiJie Chen, Zaiwen Wen

In this paper, we consider the linear programming (LP) formulation for deep reinforcement learning. The number of the constraints depends on the size of state and action spaces, which makes the problem intractable in lar…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)