Accelerating Inverse Reinforcement Learning with Expert Bootstrapping

2024-02-04 · David Wu, Sanjiban Choudhury

Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL) search over candidate reward functions and solve a reinforcement learning problem in the inner loop. This creates a rather strange inversion where a harder problem, reinforcement learning, is in the inner loop of a presumably easier problem, imitation learning. In this work, we show that better utilization of expert demonstrations can reduce the need for hard exploration in the inner RL loop, hence accelerating learning. Specifically, we propose two simple recipes: (1) placing expert transitions into the replay buffer of the inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner about high reward states instead of forcing the learner to discover them through extensive exploration, and (2) using expert actions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high value expert states. Our methods show significant gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks, speeding up recovery to 70\% of deterministic expert performance by 2.13x on HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.

📄 PDF Abstract BibTeX arXiv:2402.02608

Code (0)

등록된 구현이 없습니다.

Tasks

Imitation LearningMuJoCoreinforcement-learning Reinforcement Learning

Bootstrapping Apprenticeship Learning

2010-12-01 · NeurIPS 2010 12 · Abdeslam Boularias, Brahim Chaib-Draa

We consider the problem of apprenticeship learning where the examples, demonstrated by an expert, cover only a small part of a large state space. Inverse Reinforcement Learning (IRL) provides an efficient tool for genera…

Car Racing Reinforcement Learning

Option Compatible Reward Inverse Reinforcement Learning

2019-11-07 · Rakhoon Hwang, Hanjin Lee, Hyung Ju Hwang

Reinforcement learning in complex environments is a challenging problem. In particular, the success of reinforcement learning algorithms depends on a well-designed reward function. Inverse reinforcement learning (IRL) so…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning

A proof of imitation of Wasserstein inverse reinforcement learning for multi-objective optimization

2023-05-17 · Akira Kitaoka, Riki Eto

We prove Wasserstein inverse reinforcement learning enables the learner's reward values to imitate the expert's reward values in a finite iteration for multi-objective optimizations. Moreover, we prove Wasserstein invers…

reinforcement-learning Reinforcement Learning

Nonlinear Inverse Reinforcement Learning with Gaussian Processes

2011-12-01 · NeurIPS 2011 12 · Sergey Levine, Zoran Popovic, Vladlen Koltun

We present a probabilistic algorithm for nonlinear inverse reinforcement learning. The goal of inverse reinforcement learning is to learn the reward function in a Markov decision process from expert demonstrations. While…

Gaussian Processes reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations

2018-01-07 · ICML 2018 7 · Xingyu Wang, Diego Klabjan

This paper considers the problem of inverse reinforcement learning in zero-sum stochastic games when expert demonstrations are known to be not optimal. Compared to previous works that decouple agents in the game by assum…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)