SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization
Incorporating safety is an essential prerequisite for broadening the practical applications of reinforcement learning in real-world scenarios. To tackle this challenge, Constrained Markov Decision Processes (CMDPs) are leveraged, which introduce a distinct cost function representing safety violations. In CMDPs' settings, Lagrangian relaxation technique has been employed in previous algorithms to convert constrained optimization problems into unconstrained dual problems. However, these algorithms may inaccurately predict unsafe behavior, resulting in instability while learning the Lagrange multiplier. This study introduces a novel safe reinforcement learning algorithm, Safety Critic Policy Optimization (SCPO). In this study, we define the safety critic, a mechanism that nullifies rewards obtained through violating safety constraints. Furthermore, our theoretical analysis indicates that the proposed algorithm can automatically balance the trade-off between adhering to safety constraints and maximizing rewards. The effectiveness of the SCPO algorithm is empirically validated by benchmarking it against strong baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Benchmarkingreinforcement-learningReinforcement LearningSafe Reinforcement LearningSimilar Papers 제목 키워드 기반
Constrained Policy Optimization via Sampling-Based Weight-Space Projection
Safety-critical learning requires policies that improve performance without leaving the safe operating regime. We study constrained policy learning where model parameters must satisfy rollout-based safety constraints tha…
State-wise Constrained Policy Optimization
Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular…
Autonomous Drivingreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
Enforcing state-wise safety constraints is critical for the application of reinforcement learning (RL) in real-world problems, such as autonomous driving and robot manipulation. However, existing safe RL methods only enf…
Autonomous Drivingcontinuous-controlContinuous ControlReinforcement Learning (RL)+1Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization
Deep reinforcement learning algorithms can perform poorly in real-world tasks due to the discrepancy between source and target environments. This discrepancy is commonly viewed as the disturbance in transition dynamics. …
Deep Reinforcement LearningSafety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints, as exceeding safety violation limits can result in severe conse…
Safe Reinforcement Learning