홈 › Papers

Conservative State Value Estimation for Offline Reinforcement Learning

2023-02-14 · NeurIPS 2023 11 · Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, QIngwei Lin, Saravan Rajmohan, Thomas Moscibroda, Dongmei Zhang

Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states \emph{around} the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods.

📄 PDF Abstract BibTeX arXiv:2302.06884

Code (1)

2023annonymousauthor/csve 공식 구현 pytorch

Tasks

D4RL reinforcement-learning Reinforcement Learning

DCE: Offline Reinforcement Learning With Double Conservative Estimates

2022-09-27 · Chen Zhao, Kai Xing Huang, Chun Yuan

Offline Reinforcement Learning has attracted much interest in solving the application challenge for traditional reinforcement learning. Offline reinforcement learning uses previously-collected datasets to train agents wi…

Computational Efficiency D4RL reinforcement-learning Reinforcement Learning+1

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

2022-06-06 · Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang 외

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are gener…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1

Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning

2026-05-14 · Byeongchan Kim, Min-hwan Oh arxiv

We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($λ$) (CPQL). Our algorithm adapts the Peng's Q($λ$) (PQL) operator for conservative value estimation as an alternati…

Reinforcement Learning Offline RL

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

2023-09-22 · NeurIPS 2023 11 · Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang 외

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the actio…

counterfactual Multi-agent Reinforcement Learning Offline RL Q-Learning+2

Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning

2025-08-08 · Haohui Chen, Zhiyong Chen arxiv

Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leadi…