Generalization in offline RL: The structure is more important than the amount of pessimism

2026-07-02 · Max Weltevrede, Matthijs T. J. Spaan, Wendelin Böhmer arxiv

While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs). Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA). Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.

📄 PDF Abstract BibTeX arXiv:2607.02288

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Data Augmentation Offline RL

What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?

2023-05-30 · Rui Yang, Yong Lin, Xiaoteng Ma, Hao Hu 외

Offline goal-conditioned RL (GCRL) offers a way to train general-purpose agents from fully offline datasets. In addition to being conservative within the dataset, the generalization ability to achieve unseen goals is ano…

Imitation Learning Offline RL

The Generalization Gap in Offline Reinforcement Learning

2023-12-10 · Ishita Mediratta, Qingfei You, Minqi Jiang, Roberta Raileanu

Despite recent progress in offline learning, these methods are still trained and tested on the same environment. In this paper, we compare the generalization abilities of widely used online and offline learning methods s…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning with Causal Structured World Models

2022-06-03 · Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang 외

Model-based methods have recently shown promising for offline reinforcement learning (RL), aiming to learn good policies from historical data without interacting with the environment. Previous model-based offline RL meth…

Model-based Reinforcement Learning Offline RL reinforcement-learning Reinforcement Learning+1

Offline Reinforcement Learning with Reverse Model-based Imagination

2021-10-01 · NeurIPS 2021 12 · Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu 외

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods at…

Data Augmentation model Offline RL reinforcement-learning+2

Is Value Learning Really the Main Bottleneck in Offline RL?

2024-06-13 · Seohong Park, Kevin Frans, Sergey Levine, Aviral Kumar

While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. Howeve…

Imitation Learning Offline RL Reinforcement Learning (RL)