SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories
In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each timestep can be difficult. However, it is often feasible to collect trajectories reflecting undesirable or risky behavior, implicitly conveying the behavior the agent should avoid. We refer to these trajectories as non-preferred trajectories. Unlike standard IL, which aims to mimic demonstrations, our agent must also learn to avoid risky behavior using non-preferred trajectories. In this paper, we propose a novel approach, SafeMIL, to learn a parameterized cost that predicts if the state-action pair is risky via Multiple Instance Learning. The learned cost is then used to avoid non-preferred behaviors, resulting in a policy that prioritizes safety. We empirically demonstrate that our approach can learn a safer policy that satisfies cost constraints without degrading the reward performance, thereby outperforming several baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Multiple Instance LearningSimilar Papers 제목 키워드 기반
SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonst…
OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories
This work addresses the problem of offline safe imitation learning (IL), where the goal is to learn safe and reward-maximizing policies from demonstrations that do not have per-timestep safety cost or reward information.…
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
We address the problem of making a pre-trained reinforcement learning (RL) policy safety-aware by incorporating cost constraints without retraining it from scratch. While costs could be numerically encoded, we assume a m…
Computational EfficiencyReinforcement LearningContinuous ControlDecision MakingPreferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
Offline reinforcement learning (RL) aims to learn optimal policies from previously collected datasets. Recently, due to their powerful representational capabilities, diffusion models have shown significant potential as p…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Offline safe reinforcement learning (RL) aims to train a policy that satisfies constraints using a pre-collected dataset. Most current methods struggle with the mismatch between imperfect demonstrations and the desired s…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning