Mildly Conservative Q-Learning for Offline Reinforcement Learning

2022-06-09 · Jiafei Lyu, Xiaoteng Ma, Xiu Li, Zongqing Lu

Offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic, which suppresses the generalization of the value function and hinders the performance improvement. This paper explores mild but enough conservatism for offline learning while not harming generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values. We theoretically show that MCQ induces a policy that behaves at least as well as the behavior policy and no erroneous overestimation will occur for OOD actions. Experimental results on the D4RL benchmarks demonstrate that MCQ achieves remarkable performance compared with prior work. Furthermore, MCQ shows superior generalization ability when transferring from offline to online, and significantly outperforms baselines. Our code is publicly available at https://github.com/dmksjfl/MCQ.

📄 PDF Abstract BibTeX arXiv:2206.04745

Code (3)

dmksjfl/mcq 공식 구현 pytorch

yihaosun1124/OfflineRL-Kit pytorch

zzmtsvv/ORL pytorch

Tasks

D4RLQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning

2025-08-08 · Haohui Chen, Zhiyong Chen arxiv

Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leadi…

Reinforcement Learning Offline RL

DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

2023-09-16 · Xiao-Yin Liu, Xiao-Hu Zhou, Mei-Jiang Gui, Xiao-Liang Xie 외

Model-based reinforcement learning (RL), which learns environment model from offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in off…

D4RL model Model-based Reinforcement Learning Offline RL+3

Generalization in offline RL: The structure is more important than the amount of pessimism

2026-07-02 · Max Weltevrede, Matthijs T. J. Spaan, Wendelin Böhmer arxiv

While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate …

Reinforcement Learning Data Augmentation Offline RL

DCE: Offline Reinforcement Learning With Double Conservative Estimates

2022-09-27 · Chen Zhao, Kai Xing Huang, Chun Yuan

Offline Reinforcement Learning has attracted much interest in solving the application challenge for traditional reinforcement learning. Offline reinforcement learning uses previously-collected datasets to train agents wi…

Computational Efficiency D4RL reinforcement-learning Reinforcement Learning+1

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

2022-06-06 · Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang 외

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are gener…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1