Pretrain Soft Q-Learning with Imperfect Demonstrations

2019-05-09 · Xiaoqin Zhang, Yunfei Li, Huimin Ma, Xiong Luo

Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing reinforcement learning algorithms. Pretraining reinforcement learning remains a significant challenge in exploiting expert demonstrations whilst keeping exploration potentials, especially for value based methods. In this paper, we propose a pretraining method for soft Q-learning. Our work is inspired by pretraining methods for actor-critic algorithms since soft Q-learning is a value based algorithm that is equivalent to policy gradient. The proposed method is based on $\gamma$-discounted biased policy evaluation with entropy regularization, which is also the updating target of soft Q-learning. Our method is evaluated on various tasks from Atari 2600. Experiments show that our method effectively learns from imperfect demonstrations, and outperforms other state-of-the-art methods that learn from expert demonstrations.

📄 PDF Abstract BibTeX arXiv:1905.03501

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

2019-11-16 · Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun 외

In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations. Most of existing RLfD methods require …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Imitation Learning from Imperfect Demonstration

2019-01-27 · Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt 외

Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, w…

Imitation Learning

Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning

2023-02-13 · Yunke Wang, Bo Du, Chang Xu

Adversarial imitation learning has become a widely used imitation learning framework. The discriminator is often trained by taking expert demonstrations and policy trajectories as examples respectively from two categorie…

Imitation Learning MuJoCo

Imitation Learning from Purified Demonstrations

2023-10-11 · Yunke Wang, Minjing Dong, Yukun Zhao, Bo Du 외

Imitation learning has emerged as a promising approach for addressing sequential decision-making problems, with the assumption that expert demonstrations are optimal. However, in real-world scenarios, most demonstrations…

Decision Making Imitation Learning MuJoCo Sequential Decision Making

DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations

2021-09-29 · ICLR 2022 4 · Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon 외

We consider offline imitation learning (IL), which aims to mimic the expert's behavior from its demonstration without further interaction with the environment. One of the main challenges in offline IL is to deal with th…