Iterative Reachability Estimation for Safe Reinforcement Learning
Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSimilar Papers 제목 키워드 기반
Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey
Recent literature has proposed approaches that learn control policies with high performance while maintaining safety guarantees. Synthesizing Hamilton-Jacobi (HJ) reachable sets has become an effective tool for verifying…
reinforcement-learningReinforcement LearningSurveySafe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inferen…
Reinforcement LearningOffline RLReachability Constrained Reinforcement Learning
Constrained reinforcement learning (CRL) has gained significant interest recently, since safety constraints satisfaction is critical for real-world problems. However, existing CRL methods constraining discounted cumulati…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning$λ$-Reachability: Geometric-Horizon Safety Bellman Equations for Humanoid Safety
We introduce $λ$-Reachability, a scalable approach to Hamilton--Jacobi safety analysis for high-dimensional robotic systems. Unlike prior discounted formulations that rely on fixed one-step Bellman updates, $λ$-Reachabil…
Collision AvoidanceReinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions
Reinforcement learning (RL) exhibits impressive performance when managing complicated control tasks for robots. However, its wide application to physical robots is limited by the absence of strong safety guarantees. To o…
reinforcement-learningReinforcement Learning (RL)