Offline Learning from Demonstrations and Unlabeled Experience
Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. However, BC does not effectively leverage what we will refer to as unlabeled experience: data of mixed and unknown quality without reward annotations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. Towards data-driven offline robot learning that can use this unlabeled experience, we introduce Offline Reinforced Imitation Learning (ORIL). ORIL first learns a reward function by contrasting observations from demonstrator and unlabeled trajectories, then annotates all data with the learned reward, and finally trains an agent via offline reinforcement learning. Across a diverse set of continuous control and simulated robotic manipulation tasks, we show that ORIL consistently outperforms comparable BC agents by effectively leveraging unlabeled experience.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlImitation LearningSimilar Papers 제목 키워드 기반
SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of …
Imitation LearningQ-LearningSafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonst…
Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble
Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement l…
continuous-controlContinuous ControlDeep Reinforcement LearningImitation Learning+4UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations
We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting…
Imitation LearningQ-LearningRobust Offline Imitation Learning Through State-level Trajectory Stitching
Imitation learning (IL) has proven effective for enabling robots to acquire visuomotor skills through expert demonstrations. However, traditional IL methods are limited by their reliance on high-quality, often scarce, ex…
Imitation Learning