f-IRL: Inverse Reinforcement Learning via State Marginal Matching

2020-11-09 · Tianwei Ni, Harshit Sikchi, YuFei Wang, Tejus Gupta, Lisa Lee, Benjamin Eysenbach

Imitation learning is well-suited for robotic tasks where it is difficult to directly program the behavior or specify a cost for optimal control. In this work, we propose a method for learning the reward function (and the corresponding policy) to match the expert state density. Our main result is the analytic gradient of any f-divergence between the agent and expert state distribution w.r.t. reward parameters. Based on the derived gradient, we present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. We show that f-IRL can learn behaviors from a hand-designed target state density or implicitly through expert observations. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks. Moreover, we show that the recovered reward function can be used to quickly solve downstream tasks, and empirically demonstrate its utility on hard-to-explore tasks and for behavior transfer across changes in dynamics.

📄 PDF Abstract BibTeX arXiv:2011.04709

Code (1)

twni2016/f-IRL 공식 구현 pytorch

Tasks

Imitation Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributional Decision Transformer for Hindsight Information Matching

2021-09-29 · ICLR 2022 4 · Hiroki Furuta, Yutaka Matsuo, Shixiang Shane Gu

How to extract as much learning signal from each trajectory data has been a key problem in reinforcement learning (RL), where sample inefficiency has posed serious challenges for practical applications. Recent works have…

continuous-control Continuous Control Imitation Learning MuJoCo+2

A Divergence Minimization Perspective on Imitation Learning Methods

2019-11-06 · Seyed Kamyar Seyed Ghasemipour, Richard Zemel, Shixiang Gu

In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Be…

Behavioural cloning continuous-control Continuous Control Decision Making+2

Learning to Match via Inverse Optimal Transport

2018-02-10 · Ruilin Li, Xiaojing Ye, Haomin Zhou, Hongyuan Zha

We propose a unified data-driven framework based on inverse optimal transport that can learn adaptive, nonlinear interaction cost function from noisy and incomplete empirical matching matrix and predict new matching in v…

Efficient Exploration via State Marginal Matching

2019-06-12 · Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing 외

Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mat…

Efficient Exploration Reinforcement Learning Unsupervised Reinforcement Learning

How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation?

2021-08-01 · ACL 2021 5 · Sayan Ghosh, Zheng Qi, Snigdha Chaturvedi, Shashank Srivastava

Existing approaches for the Table-to-Text task suffer from issues such as missing information, hallucination and repetition. Many approaches to this problem use Reinforcement Learning (RL), which maximizes a single manua…

Domain Adaptation Hallucination reinforcement-learning Reinforcement Learning+3