Learning from Sparse Offline Datasets via Conservative Density Estimation

2024-01-16 · Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, Ding Zhao

Offline reinforcement learning (RL) offers a promising direction for learning policies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle to handle out-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose a novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing constraints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correction method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-art performance on the D4RL benchmark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem in offline RL.

📄 PDF Abstract BibTeX arXiv:2401.08819

Code (1)

czp16/cde-offline-rl 공식 구현 pytorch

Tasks

D4RL Density Estimation Offline RL Reinforcement Learning (RL)

Generative OOD-regularized Model-based Policy Optimization

2026-05-23 · Aysin Tumay, Jiahe Huang, Elise Jortberg, Rose Yu arxiv

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. T…

Reinforcement Learning Density Estimation Offline RL

DCE: Offline Reinforcement Learning With Double Conservative Estimates

2022-09-27 · Chen Zhao, Kai Xing Huang, Chun Yuan

Offline Reinforcement Learning has attracted much interest in solving the application challenge for traditional reinforcement learning. Offline reinforcement learning uses previously-collected datasets to train agents wi…

Computational Efficiency D4RL reinforcement-learning Reinforcement Learning+1

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

2024-06-11 · Zeyuan Liu, Kai Yang, Xiu Li

Distribution shift is a major obstacle in offline reinforcement learning, which necessitates minimizing the discrepancy between the learned policy and the behavior policy to avoid overestimating rare or unseen actions. P…

D4RL Denoising Offline RL reinforcement-learning+1

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

2023-09-22 · NeurIPS 2023 11 · Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang 외

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the actio…

counterfactual Multi-agent Reinforcement Learning Offline RL Q-Learning+2

Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning

2025-08-08 · Haohui Chen, Zhiyong Chen arxiv

Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leadi…

Reinforcement Learning Offline RL