Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

2024-04-06 · Yeda Song, Dongwook Lee, Gunhee Kim

Offline reinforcement learning (RL) is a compelling framework for learning optimal policies from past experiences without additional interaction with the environment. Nevertheless, offline RL inevitably faces the problem of distributional shifts, where the states and actions encountered during policy execution may not be in the training dataset distribution. A common solution involves incorporating conservatism into the policy or the value function to safeguard against uncertainties and unknowns. In this work, we focus on achieving the same objectives of conservatism but from a different perspective. We propose COmpositional COnservatism with Anchor-seeking (COCOA) for offline RL, an approach that pursues conservatism in a compositional manner on top of the transductive reparameterization (Netanyahu et al., 2023), which decomposes the input variable (the state in our case) into an anchor and its difference from the original input. Our COCOA seeks both in-distribution anchors and differences by utilizing the learned reverse dynamics model, encouraging conservatism in the compositional input space for the policy or value function. Such compositional conservatism is independent of and agnostic to the prevalent behavioral conservatism in offline RL. We apply COCOA to four state-of-the-art offline RL algorithms and evaluate them on the D4RL benchmark, where COCOA generally improves the performance of each algorithm. The code is available at https://github.com/runamu/compositional-conservatism.

📄 PDF Abstract BibTeX arXiv:2404.04682

Code (1)

runamu/compositional-conservatism 공식 구현 pytorch

Tasks

D4RL Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Focus 설명 없음

UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

2026-05-28 · Aditya Upadhyay arxiv

Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ …

Reinforcement Learning Offline RL

Offline Reinforcement Learning with Reverse Model-based Imagination

2021-10-01 · NeurIPS 2021 12 · Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu 외

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods at…

Data Augmentation model Offline RL reinforcement-learning+2

Model-based Offline Reinforcement Learning with Count-based Conservatism

2023-07-21 · Byeongchan Kim, Min-hwan Oh

In this paper, we propose a model-based offline reinforcement learning method that integrates count-based conservatism, named $\texttt{Count-MORL}$. Our method utilizes the count estimates of state-action pairs to quanti…

D4RL Offline RL reinforcement-learning Reinforcement Learning

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

2022-06-06 · Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang 외

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are gener…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

2025-12-04 · Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga 외 arxiv

Popular offline reinforcement learning (RL) methods rely on explicit conservatism, penalizing out-of-dataset actions or restricting rollout horizons. We question the universality of this principle and revisit a complemen…

Reinforcement Learning Test-time Adaptation