Imitation Learning by Estimating Expertise of Demonstrators

2022-02-02 · Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, Ramtin Pedarsani

Many existing imitation learning datasets are collected from multiple demonstrators, each with different expertise at different parts of the environment. Yet, standard imitation learning algorithms typically treat all demonstrators as homogeneous, regardless of their expertise, absorbing the weaknesses of any suboptimal demonstrators. In this work, we show that unsupervised learning over demonstrator expertise can lead to a consistent boost in the performance of imitation learning algorithms. We develop and optimize a joint model over a learned policy and expertise levels of the demonstrators. This enables our model to learn from the optimal behavior and filter out the suboptimal behavior of each demonstrator. Our model learns a single policy that can outperform even the best demonstrator, and can be used to estimate the expertise of any demonstrator at any state. We illustrate our findings on real-robotic continuous control tasks from Robomimic and discrete environments such as MiniGrid and chess, out-performing competing methods in $21$ out of $23$ settings, with an average of $7\%$ and up to $60\%$ improvement in terms of the final reward.

📄 PDF Abstract BibTeX arXiv:2202.01288

Code (1)

stanford-iliad/ileed 공식 구현 pytorch

Tasks

continuous-control Continuous ControlImitation Learning

Inverse Reinforcement Learning by Estimating Expertise of Demonstrators

2024-02-02 · Mark Beliaev, Ramtin Pedarsani

In Imitation Learning (IL), utilizing suboptimal and heterogeneous demonstrations presents a substantial challenge due to the varied nature of real-world data. However, standard IL algorithms consider these datasets as h…

Imitation Learning reinforcement-learning Reinforcement Learning

VILD: Variational Imitation Learning with Diverse-quality Demonstrations

2019-09-15 · Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, Masashi Sugiyama

The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations f…

continuous-control Continuous Control Imitation Learning Reinforcement Learning

ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games

2024-02-28 · Shiqi Lei, Kanghoon Lee, Linjing Li, Jinkyoo Park 외

Offline learning has become widely used due to its ability to derive effective policies from offline datasets gathered by expert demonstrators without interacting with the environment directly. Recent research has explor…

Imitation Learning

Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations

2023-10-24 · Ye Yuan, Xin Li, Yong Heng, Leiji Zhang 외

Imitation Learning (IL) aims to discover a policy by minimizing the discrepancy between the agent's behavior and expert demonstrations. However, IL is susceptible to limitations imposed by noisy demonstrations from non-e…

Imitation Learning MuJoCo

Inverse Reinforcement Learning with Multiple Ranked Experts

2019-07-31 · Pablo Samuel Castro, Shijian Li, Daqing Zhang

We consider the problem of learning to behave optimally in a Markov Decision Process when a reward function is not specified, but instead we have access to a set of demonstrators of varying performance. We assume the dem…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)