Optimal Transport for Offline Imitation Learning
With the advent of large datasets, offline reinforcement learning (RL) is a promising framework for learning good decision-making policies without the need to interact with the real environment. However, offline RL requires the dataset to be reward-annotated, which presents practical challenges when reward engineering is difficult or when obtaining reward annotations is labor-intensive. In this paper, we introduce Optimal Transport Reward labeling (OTR), an algorithm that assigns rewards to offline trajectories, with a few high-quality demonstrations. OTR's key idea is to use optimal transport to compute an optimal alignment between an unlabeled trajectory in the dataset and an expert demonstration to obtain a similarity measure that can be interpreted as a reward, which can then be used by an offline RL algorithm to learn the policy. OTR is easy to implement and computationally efficient. On D4RL benchmarks, we show that OTR with a single demonstration can consistently match the performance of offline RL with ground-truth rewards.
Code (1)
Tasks
D4RLDecision MakingImitation LearningOffline RLReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Align Your Intents: Offline Imitation Learning via Optimal Transport
Offline Reinforcement Learning (RL) addresses the problem of sequential decision-making by learning optimal policy through pre-collected data, without interacting with the environment. As yet, it has remained somewhat im…
D4RLDecision MakingImitation LearningOffline RL+2Rethinking Optimal Transport in Offline Reinforcement Learning
We propose a novel algorithm for offline reinforcement learning using optimal transport. Typically, in offline reinforcement learning, the data is provided by various experts and some of them can be sub-optimal. To extra…
continuous-controlContinuous ControlD4RLreinforcement-learning+1Zero-Shot Offline Imitation Learning via Optimal Transport
Zero-shot imitation learning algorithms hold the promise of reproducing unseen behavior from as little as a single demonstration at test time. Existing practical approaches view the expert demonstration as a sequence of …
Imitation LearningLeveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environments
Most Reinforcement Learning (RL) methods are traditionally studied in an active learning setting, where agents directly interact with their environments, observe action outcomes, and learn through trial and error. Howeve…
Active LearningOffline RLReinforcement Learning (RL)Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback,…
Reinforcement Learning