Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

2019-11-16 · Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, Huaping Liu

In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations. Most of existing RLfD methods require demonstrations to be perfect and sufficient, which yet is unrealistic to meet in practice. To work on imperfect demonstrations, we first define an imperfect expert setting for RLfD in a formal way, and then point out that previous methods suffer from two issues in terms of optimality and convergence, respectively. Upon the theoretical findings we have derived, we tackle these two issues by regarding the expert guidance as a soft constraint on regulating the policy exploration of the agent, which eventually leads to a constrained optimization problem. We further demonstrate that such problem is able to be addressed efficiently by performing a local linear search on its dual form. Considerable empirical evaluations on a comprehensive collection of benchmarks indicate our method attains consistent improvement over other RLfD counterparts.

📄 PDF Abstract BibTeX arXiv:1911.07109

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Pretrain Soft Q-Learning with Imperfect Demonstrations

2019-05-09 · Xiaoqin Zhang, Yunfei Li, Huimin Ma, Xiong Luo

Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing rei…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Bayesian Q-learning With Imperfect Expert Demonstrations

2022-10-01 · Fengdi Che, Xiru Zhu, Doina Precup, David Meger 외

Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expert information. We propose a novel algorithm to speed up Q-learning with the hel…

Atari Games Q-Learning reinforcement-learning Reinforcement Learning (RL)

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble

2022-12-07 · Chao Li

Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement l…

continuous-control Continuous Control Deep Reinforcement Learning Imitation Learning+4

Hybrid Policy Optimization from Imperfect Demonstrations

2023-09-21 · NeurIPS 2023 11

Exploration is one of the main challenges in Reinforcement Learning (RL), especially in environments with sparse rewards. Learning from Demonstrations (LfD) is a promising approach to solving this problem by leveraging e…

DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations

2021-09-29 · ICLR 2022 4 · Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon 외

We consider offline imitation learning (IL), which aims to mimic the expert's behavior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with th…

Imitation Learning