Model-based Offline Reinforcement Learning with Count-based Conservatism

2023-07-21 · Byeongchan Kim, Min-hwan Oh

In this paper, we propose a model-based offline reinforcement learning method that integrates count-based conservatism, named $\texttt{Count-MORL}$. Our method utilizes the count estimates of state-action pairs to quantify model estimation error, marking the first algorithm of demonstrating the efficacy of count-based conservatism in model-based offline deep RL to the best of our knowledge. For our proposed method, we first show that the estimation error is inversely proportional to the frequency of state-action pairs. Secondly, we demonstrate that the learned policy under the count-based conservative model offers near-optimality performance guarantees. Through extensive numerical experiments, we validate that $\texttt{Count-MORL}$ with hash code implementation significantly outperforms existing offline RL algorithms on the D4RL benchmark datasets. The code is accessible at $\href{https://github.com/oh-lab/Count-MORL}{https://github.com/oh-lab/Count-MORL}$.

📄 PDF Abstract BibTeX arXiv:2307.11352

Code (1)

oh-lab/count-morl 공식 구현 pytorch

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning

Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

2024-04-06 · Yeda Song, Dongwook Lee, Gunhee Kim

Offline reinforcement learning (RL) is a compelling framework for learning optimal policies from past experiences without additional interaction with the environment. Nevertheless, offline RL inevitably faces the problem…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

2022-06-06 · Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang 외

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are gener…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1

UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

2026-05-28 · Aditya Upadhyay arxiv

Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ …

Reinforcement Learning Offline RL

Offline Reinforcement Learning with Reverse Model-based Imagination

2021-10-01 · NeurIPS 2021 12 · Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu 외

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods at…

Data Augmentation model Offline RL reinforcement-learning+2

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

2025-12-04 · Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga 외 arxiv

Popular offline reinforcement learning (RL) methods rely on explicit conservatism, penalizing out-of-dataset actions or restricting rollout horizons. We question the universality of this principle and revisit a complemen…

Reinforcement Learning Test-time Adaptation