Imitation Learning as $f$-Divergence Minimization
We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any f-Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and Dagger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningSimilar Papers 제목 키워드 기반
Reparameterized Variational Divergence Minimization for Stable Imitation
While recent state-of-the-art results for adversarial imitation-learning algorithms are encouraging, recent works exploring the imitation learning from observation (ILO) setting, where trajectories \textit{only} contain …
continuous-controlContinuous ControlImitation LearningA Divergence Minimization Perspective on Imitation Learning Methods
In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Be…
Behavioural cloningcontinuous-controlContinuous ControlDecision Making+2Modeling Strong and Human-Like Gameplay with KL-Regularized Search
We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior. Imitation learning is effective at predicting human actions but may not match the…
Decision MakingImitation LearningImitation Learning with Sinkhorn Distances
Imitation learning algorithms have been interpreted as variants of divergence minimization problems. The ability to compare occupancy measures between experts and learners is crucial in their effectiveness in learning fr…
Imitation LearningMuJoCoEquivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences
The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solu…
Inductive Bias