Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples

2021-06-28 · Li Meng, Anis Yazidi, Morten Goodwin, Paal Engelstad

In this article, we propose a novel algorithm for deep reinforcement learning named Expert Q-learning. Expert Q-learning is inspired by Dueling Q-learning and aims at incorporating semi-supervised learning into reinforcement learning through splitting Q-values into state values and action advantages. We require that an offline expert assesses the value of a state in a coarse manner using three discrete values. An expert network is designed in addition to the Q-network, which updates each time following the regular offline minibatch update whenever the expert example buffer is not empty. Using the board game Othello, we compare our algorithm with the baseline Q-learning algorithm, which is a combination of Double Q-learning and Dueling Q-learning. Our results show that Expert Q-learning is indeed useful and more resistant to the overestimation bias. The baseline Q-learning algorithm exhibits unstable and suboptimal behavior in non-deterministic settings, whereas Expert Q-learning demonstrates more robust performance with higher scores, illustrating that our algorithm is indeed suitable to integrate state values from expert examples into Q-learning.

📄 PDF Abstract BibTeX arXiv:2106.14642

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningImitation LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Double Q-learning Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. The…

BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning

2025-11-27 · Junsung Park arxiv

Offline inverse reinforcement learning (IRL) aims to recover a reward function that explains expert behavior using only fixed demonstration data, without any additional online interaction. We propose BiCQL-ML, a policy-f…

Reinforcement Learning Offline RL

Rethinking Optimal Transport in Offline Reinforcement Learning

2024-10-17 · Arip Asadulaev, Rostislav Korst, Alexander Korotin, Vage Egiazarian 외

We propose a novel algorithm for offline reinforcement learning using optimal transport. Typically, in offline reinforcement learning, the data is provided by various experts and some of them can be sub-optimal. To extra…

continuous-control Continuous Control D4RL reinforcement-learning+1

The Virtues of Pessimism in Inverse Reinforcement Learning

2024-02-04 · David Wu, Gokul Swamy, J. Andrew Bagnell, Zhiwei Steven Wu 외

Inverse Reinforcement Learning (IRL) is a powerful framework for learning complex behaviors from expert demonstrations. However, it traditionally requires repeatedly solving a computationally expensive reinforcement lear…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

SAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance

2024-10-24 · Liyu Zhang, Haochi Wu, Xu Wan, Quan Kong 외

The offline-to-online (O2O) paradigm in reinforcement learning (RL) utilizes pre-trained models on offline datasets for subsequent online fine-tuning. However, conventional O2O RL algorithms typically require maintaining…

D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network

2025-02-01 · Jijia Liu, Feng Gao, Qingmin Liao, Chao Yu 외

Reinforcement learning (RL) for continuous control often requires large amounts of online interaction data. Value-based RL methods can mitigate this burden by offering relatively high sample efficiency. Some studies furt…

continuous-control Continuous Control D4RL Q-Learning+1