Effects of Conservatism on Offline Learning

2021-09-29 · Karush Suri, Florian Shkurti

Conservatism, the act of underestimating an agent's expected value estimates, has demonstrated profound success in model-free, model-based, multi-task, safe and other realms of offline Reinforcement Learning (RL). Recent work, on the other hand, has noted that conservatism often hinders learning of behaviors. To that end, the paper asks the question how does conservatism affect offline learning? The proposed answer studies conservatism in light of value function optimization, approximate objectives that upper bound underestimations and behavior cloning as auxilary regularization objective. Conservative agents implicitly steer estimates away from the true value function, resulting in optimization objectives with high condition numbers. Mitigating these issues requires an upper bounding objective. These approximate upper bounds, however, impose strong geometrical assumptions on the dataset design, a result which is only sparsely fulfilled. Driven by theoretical observations, provision of an auxilary behavior cloning objective as variational regularization to estimates results in accurate value estimation, well-conditioned search spaces and expressive parameterizations. In an empirical study of discrete and continuous control tasks, we validate our theoretical insights and demonstrate the practical effects of learning underestimated value functions.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control Reinforcement Learning (RL)

Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

2024-04-06 · Yeda Song, Dongwook Lee, Gunhee Kim

Offline reinforcement learning (RL) is a compelling framework for learning optimal policies from past experiences without additional interaction with the environment. Nevertheless, offline RL inevitably faces the problem…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

Offline Reinforcement Learning with Reverse Model-based Imagination

2021-10-01 · NeurIPS 2021 12 · Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu 외

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods at…

Data Augmentation model Offline RL reinforcement-learning+2

UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

2026-05-28 · Aditya Upadhyay arxiv

Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ …

Reinforcement Learning Offline RL

Model-based Offline Reinforcement Learning with Count-based Conservatism

2023-07-21 · Byeongchan Kim, Min-hwan Oh

In this paper, we propose a model-based offline reinforcement learning method that integrates count-based conservatism, named $\texttt{Count-MORL}$. Our method utilizes the count estimates of state-action pairs to quanti…

D4RL Offline RL reinforcement-learning Reinforcement Learning

DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

2023-09-16 · Xiao-Yin Liu, Xiao-Hu Zhou, Mei-Jiang Gui, Xiao-Liang Xie 외

Model-based reinforcement learning (RL), which learns environment model from offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in off…

D4RL model Model-based Reinforcement Learning Offline RL+3