SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonstrations. This problem setting corresponds to various real-world scenarios, where satisfying safety constraints is more important than maximizing the expected return. However, it is very challenging to learn the policy to avoid constraint-violating (i.e. non-preferred) behavior, as opposed to standard imitation learning which learns the policy to mimic given demonstrations. In this paper, we present a hyperparameter-free offline safe IL algorithm, SafeDICE, that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algorithm directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. In the experiments, we demonstrate that our algorithm learns a more safe policy that satisfies the cost constraint without degrading the reward performance, compared to baseline algorithms.Submission Number: 13161
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
This work addresses the problem of offline safe imitation learning (IL), where the goal is to learn safe and reward-maximizing policies from demonstrations that do not have per-timestep safety cost or reward information.…
SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories
In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each time…
Multiple Instance LearningOASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Offline safe reinforcement learning (RL) aims to train a policy that satisfies constraints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired s…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSafe Reinforcement Learning with Minimal Supervision
Reinforcement learning (RL) in the real world necessitates the development of procedures that enable agents to explore without causing harm to themselves or others. The most successful solutions to the problem of safe RL…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningOffline Imitation Learning with Model-based Reverse Augmentation
In offline Imitation Learning (IL), one of the main challenges is the \textit{covariate shift} between the expert observations and the actual distribution encountered by the agent, because it is difficult to determine wh…
Imitation Learningmodel