Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning
Safe reinforcement learning (safe RL) aims to respect safety requirements while optimizing long-term performance. In many practical applications, however, the problem involves an infinite number of constraints, known as semi-infinite safe RL (SI-safe RL). Such constraints typically appear when safety conditions must be enforced across an entire continuous parameter space, such as ensuring adequate resource distribution at every spatial location. In this paper, we propose exchange policy optimization (EPO), an algorithmic framework that achieves optimal policy performance and deterministic bounded safety. EPO works by iteratively solving safe RL subproblems with finite constraint sets and adaptively adjusting the active set through constraint expansion and deletion. At each iteration, constraints with violations exceeding the predefined tolerance are added to refine the policy, while those with zero Lagrange multipliers are removed after the policy update. This exchange rule prevents uncontrolled growth of the working set and supports effective policy training. Our theoretical analysis demonstrates that, under mild assumptions, strategies trained via EPO achieve performance comparable to optimal solutions with global constraint violations strictly remaining within a prescribed bound.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning
We propose a novel generalization of constrained Markov decision processes (CMDPs) that we call the \emph{semi-infinitely constrained Markov decision process} (SICMDP). Particularly, we consider a continuum of constraint…
Decision MakingDeep Reinforcement LearningModel-based Reinforcement Learningreinforcement-learning+2From Semi-Infinite Constraints to Structured Robust Policies: Optimal Gain Selection for Financial Systems
This paper studies the robust optimal gain selection problem for financial trading systems, formulated within a \emph{double linear policy} framework, which allocates capital across long and short positions. The key obje…
Batch Policy Learning in Average Reward Markov Decision Processes
We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward…
Processing Network Controls via Deep Reinforcement Learning
Novel advanced policy gradient (APG) algorithms, such as proximal policy optimization (PPO), trust region policy optimization, and their variations, have become the dominant reinforcement learning (RL) algorithms because…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Convergence for Natural Policy Gradient on Infinite-State Queueing MDPs
A wide variety of queueing systems can be naturally modeled as infinite-state Markov Decision Processes (MDPs). In the reinforcement learning (RL) context, a variety of algorithms have been developed to learn and optimiz…
Reinforcement Learning (RL)