VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

2025-04-16 · Xuyang Chen, GuoJian Wang, Keyu Yan, Lin Zhao

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. It offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. As a result, VIPO achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks.

📄 PDF Abstract BibTeX arXiv:2504.11944

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Exclusively Penalized Q-learning for Offline Reinforcement Learning

2024-05-23 · Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee 외

Constraint-based offline reinforcement learning (RL) involves policy constraints or imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on a limit…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning

2023-07-01 · PMLR 2023 7 · Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin 외

For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learnin…

D4RL model Offline RL reinforcement-learning+3

Value Penalized Q-Learning for Recommender Systems

2021-10-15 · Chengqian Gao, Ke Xu, Kuangqi Zhou, Lanqing Li 외

Scaling reinforcement learning (RL) to recommender systems (RS) is promising since maximizing the expected cumulative rewards for RL agents meets the objective of RS, i.e., improving customers' long-term satisfaction. A …

Offline RL Q-Learning Recommendation Systems Reinforcement Learning (RL)

Critic-Guided Decision Transformer for Offline Reinforcement Learning

2023-12-21 · Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu 외

Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for ea…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning

2021-07-19 · Haoran Xu, Xianyuan Zhan, Xiangyu Zhu

We study the problem of safe offline reinforcement learning (RL), the goal is to learn a policy that maximizes long-term reward while satisfying safety constraints given only offline data, without further interaction wit…