Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations

2023-08-07 · Nirbhay Modhe, Qiaozi Gao, Ashwin Kalyan, Dhruv Batra, Govind Thattai, Gaurav Sukhatme

Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions. Model-free methods penalize values at all unseen actions, while model-based methods are able to further exploit unseen states via model rollouts. However, such methods are handicapped in their ability to find unseen states far away from the available offline data due to two factors -- (a) very short rollout horizons in models due to cascading model errors, and (b) model rollouts originating solely from states observed in offline data. We relax the second assumption and present a novel unseen state augmentation strategy to allow exploitation of unseen states where the learned model and value estimates generalize. Our strategy finds unseen states by value-informed perturbations of seen states followed by filtering out states with epistemic uncertainty estimates too high (high error) or too low (too similar to seen data). We observe improved performance in several offline RL tasks and find that our augmentation strategy consistently leads to overall lower average dataset Q-value estimates i.e. more conservative Q-value estimates than a baseline.

📄 PDF Abstract BibTeX arXiv:2308.03882

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning

2022-09-30 · Daesol Cho, Dongseok Shim, H. Jin Kim

Offline reinforcement learning (Offline RL) suffers from the innate distributional shift as it cannot interact with the physical environment during training. To alleviate such limitation, state-based offline RL leverages…

Data Augmentation Image Generation Offline RL reinforcement-learning+1

Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning

2026-05-30 · Fuyuan Qian, Menglong Zhang, Song Wang, Quanying Liu arxiv

Offline meta-reinforcement learning leverages static datasets to enable agents to generalize to unseen environments by combining offline efficiency with meta-learning adaptability, yet it faces key challenges from contex…

Representation Learning Reinforcement Learning

Offline Trajectory Generalization for Offline Reinforcement Learning

2024-04-16 · Ziqi Zhao, Zhaochun Ren, Liu Yang, Fajie Yuan 외

Offline reinforcement learning (RL) aims to learn policies from static datasets of previously collected trajectories. Existing methods for offline RL either constrain the learned policy to the support of offline data or …

D4RL Data Augmentation Offline RL reinforcement-learning+2

Offline Reinforcement Learning with Implicit Q-Learning

2021-10-12 · Ilya Kostrikov, Ashvin Nair, Sergey Levine

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the beha…

D4RL Offline RL Q-Learning reinforcement-learning+2

LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning

2025-08-30 · Hanping Zhang, Yuhong Guo arxiv

Reinforcement Learning (RL) is known for its strong decision-making capabilities and has been widely applied in various real-world scenarios. However, with the increasing availability of offline datasets and the lack of …

Natural Language Understanding Reinforcement Learning Offline RL