Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimization
Reinforcement learning (RL) has achieved promising results on most robotic control tasks. Safety of learning-based controllers is an essential notion of ensuring the effectiveness of the controllers. Current methods adopt whole consistency constraints during the training, thus resulting in inefficient exploration in the early stage. In this paper, we propose an algorithm named Constrained Policy Optimization with Extra Safety Budget (ESB-CPO) to strike a balance between the exploration efficiency and the constraints satisfaction. In the early stage, our method loosens the practical constraints of unsafe transitions (adding extra safety budget) with the aid of a new metric we propose. With the training process, the constraints in our optimization problem become tighter. Meanwhile, theoretical analysis and practical experiments demonstrate that our method gradually meets the cost limit's demand in the final training stage. When evaluated on Safety-Gym and Bullet-Safety-Gym benchmarks, our method has shown its advantages over baseline algorithms in terms of safety and optimality. Remarkably, our method gains remarkable performance improvement under the same cost limit compared with baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Efficient ExplorationReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Trajectory Planning for Safe Dual Control with Active Exploration
Planning safe trajectories under model uncertainty is a fundamental challenge. Robust planning ensures safety by considering worst-case realizations, yet ignores uncertainty reduction and leads to overly conservative beh…
Trajectory PlanningAdversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
Constrained reinforcement learning has achieved promising progress in safety-critical fields where both rewards and constraints are considered. However, constrained reinforcement learning methods face challenges in strik…
reinforcement-learningReinforcement LearningOff-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL m…
Reinforcement LearningAutonomous DrivingA Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments
Training a model-free reinforcement learning agent requires allowing the agent to sufficiently explore the environment to search for an optimal policy. In safety-constrained environments, utilizing unsupervised explorati…
Binary ClassificationSafe ExplorationImagine to Ensure Safety in Hierarchical Reinforcement Learning
This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes even more pron…
Hierarchical Reinforcement Learning