Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

2024-10-27 · Jing Zhang, Linjiajie Fang, Kexin Shi, Wenjia Wang, Bing-Yi Jing

``Distribution shift'' is the main obstacle to the success of offline reinforcement learning. A learning policy may take actions beyond the behavior policy's knowledge, referred to as Out-of-Distribution (OOD) actions. The Q-values for these OOD actions can be easily overestimated. As a result, the learning policy is biased by using incorrect Q-value estimates. One common approach to avoid Q-value overestimation is to make a pessimistic adjustment. Our key idea is to penalize the Q-values of OOD actions associated with high uncertainty. In this work, we propose Q-Distribution Guided Q-Learning (QDQ), which applies a pessimistic adjustment to Q-values in OOD regions based on uncertainty estimation. This uncertainty measure relies on the conditional Q-value distribution, learned through a high-fidelity and efficient consistency model. Additionally, to prevent overly conservative estimates, we introduce an uncertainty-aware optimization objective for updating the Q-value function. The proposed QDQ demonstrates solid theoretical guarantees for the accuracy of Q-value distribution learning and uncertainty measurement, as well as the performance of the learning policy. QDQ consistently shows strong performance on the D4RL benchmark and achieves significant improvements across many tasks.

📄 PDF Abstract BibTeX arXiv:2410.20312

Code (1)

evalarzj/qdq 공식 구현 jax

Tasks

D4RLQ-Learning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning

2026-04-10 · Mintae Kim, Koushil Sreenath arxiv

Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typicall…

Reinforcement Learning Continuous Control

Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning

2023-07-01 · PMLR 2023 7 · Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin 외

For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learnin…

D4RL model Offline RL reinforcement-learning+3

Value Penalized Q-Learning for Recommender Systems

2021-10-15 · Chengqian Gao, Ke Xu, Kuangqi Zhou, Lanqing Li 외

Scaling reinforcement learning (RL) to recommender systems (RS) is promising since maximizing the expected cumulative rewards for RL agents meets the objective of RS, i.e., improving customers' long-term satisfaction. A …

Offline RL Q-Learning Recommendation Systems Reinforcement Learning (RL)

Exclusively Penalized Q-learning for Offline Reinforcement Learning

2024-05-23 · Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee 외

Constraint-based offline reinforcement learning (RL) involves policy constraints or imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on a limit…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

2025-04-16 · Xuyang Chen, GuoJian Wang, Keyu Yan, Lin Zhao

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particul…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1