Learning from Interventions using Hierarchical Policies for Safe Learning
Learning from Demonstrations (LfD) via Behavior Cloning (BC) works well on multiple complex tasks. However, a limitation of the typical LfD approach is that it requires expert demonstrations for all scenarios, including those in which the algorithm is already well-trained. The recently proposed Learning from Interventions (LfI) overcomes this limitation by using an expert overseer. The expert overseer only intervenes when it suspects that an unsafe action is about to be taken. Although LfI significantly improves over LfD, the state-of-the-art LfI fails to account for delay caused by the expert's reaction time and only learns short-term behavior. We address these limitations by 1) interpolating the expert's interventions back in time, and 2) by splitting the policy into two hierarchical levels, one that generates sub-goals for the future and another that generates actions to reach those desired sub-goals. This sub-goal prediction forces the algorithm to learn long-term behavior while also being robust to the expert's reaction time. Our experiments show that LfI using sub-goals in a hierarchical policy framework trains faster and achieves better asymptotic performance than typical LfD.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigation
Achieving safe autonomous navigation systems is critical for deploying robots in dynamic and uncertain real-world environments. In this paper, we propose a hierarchical control framework leveraging neural network verific…
Autonomous NavigationSafe Reinforcement LearningDynamic Model Predictive Shielding for Provably Safe Reinforcement Learning
Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a backup policy to ensure safe…
reinforcement-learningReinforcement LearningSafe Reinforcement LearningHierarchical Policy Blending as Inference for Reactive Robot Control
Motion generation in cluttered, dense, and dynamic environments is a central topic in robotics, rendered as a multi-objective decision-making problem. Current approaches trade-off between safety and performance. On the o…
Decision MakingMotion GenerationStochastic OptimizationReal-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning
Foundation models can provide robust high-level reasoning on appropriate safety interventions in hazardous scenarios beyond a robot's training data, i.e. out-of-distribution (OOD) failures. However, due to the high infer…
OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation
While reinforcement learning (RL) enables robots to acquire skills autonomously, its real-world deployment is severely limited by inefficient and unsafe exploration. Human-in-the-loop interventions offer a practical solu…
Reinforcement LearningRobot Manipulation