Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
Constrained reinforcement learning has achieved promising progress in safety-critical fields where both rewards and constraints are considered. However, constrained reinforcement learning methods face challenges in striking the right balance between task performance and constraint satisfaction and it is prone for them to get stuck in over-conservative or constraint violating local minima. In this paper, we propose Adversarial Constrained Policy Optimization (ACPO), which enables simultaneous optimization of reward and the adaptation of cost budgets during training. Our approach divides original constrained problem into two adversarial stages that are solved alternately, and the policy update performance of our algorithm can be theoretically guaranteed. We validate our method through experiments conducted on Safety Gymnasium and quadruped locomotion tasks. Results demonstrate that our algorithm achieves better performances compared to commonly used baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learning
The proliferation of the Internet has led to the emergence of online advertising, driven by the mechanics of online auctions. In these repeated auctions, software agents participate on behalf of aggregated advertisers to…
Meta-Learningreinforcement-learningRobust Lagrangian and Adversarial Policy Gradient for Robust Constrained Markov Decision Processes
The robust constrained Markov decision process (RCMDP) is a recent task-modelling framework for reinforcement learning that incorporates behavioural constraints and that provides robustness to errors in the transition dy…
Incremental LearningProvably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning
We examine online safe multi-agent reinforcement learning using constrained Markov games in which agents compete by maximizing their expected total rewards under a constraint on expected total utilities. Our focus is con…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningSafe Reinforcement LearningConvergent Policy Optimization for Safe Reinforcement Learning
We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonco…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Reward Constrained Policy Optimization
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwant…
Reinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning