Offline RL with Observation Histories: Analyzing and Improving Sample Complexity

2023-10-31 · Joey Hong, Anca Dragan, Sergey Levine

Offline reinforcement learning (RL) can in principle synthesize more optimal behavior from a dataset consisting only of suboptimal trials. One way that this can happen is by "stitching" together the best parts of otherwise suboptimal trajectories that overlap on similar states, to create new behaviors where each individual state is in-distribution, but the overall returns are higher. However, in many interesting and complex applications, such as autonomous navigation and dialogue systems, the state is partially observed. Even worse, the state representation is unknown or not easy to define. In such cases, policies and value functions are often conditioned on observation histories instead of states. In these cases, it is not clear if the same kind of "stitching" is feasible at the level of observation histories, since two different trajectories would always have different histories, and thus "similar states" that might lead to effective stitching cannot be leveraged. Theoretically, we show that standard offline RL algorithms conditioned on observation histories suffer from poor sample complexity, in accordance with the above intuition. We then identify sufficient conditions under which offline RL can still be efficient -- intuitively, it needs to learn a compact representation of history comprising only features relevant for action selection. We introduce a bisimulation loss that captures the extent to which this happens, and propose that offline RL can explicitly optimize this loss to aid worst-case sample complexity. Empirically, we show that across a variety of tasks either our proposed loss improves performance, or the value of this loss is already minimized as a consequence of standard offline RL, indicating that it correlates well with good performance.

📄 PDF Abstract BibTeX arXiv:2310.20663

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous NavigationOffline RL Reinforcement Learning (RL)

Partially Observable Reinforcement Learning with Memory Traces

2025-03-19 · Onno Eberhard, Michael Muehlebach, Claire Vernade

Partially observable environments present a considerable computational challenge in reinforcement learning due to the need to consider long histories. Learning with a finite window of observations quickly becomes intract…

Partially Observable Reinforcement Learning reinforcement-learning Reinforcement Learning Value prediction

Regularized Behavior Cloning for Blocking the Leakage of Past Action Information

2023-09-21 · NeurIPS 2023 11

For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information is sufficiently captured in the observation histories for imitating the expert actions…

Minimax Optimal Q Learning with Nearest Neighbors

2023-08-03 · Puning Zhao, Lifeng Lai

Analyzing the Markov decision process (MDP) with continuous state spaces is generally challenging. A recent interesting work \cite{shah2018q} solves MDP with bounded continuous state space by a nearest neighbor $Q$ learn…

Q-Learning

Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency

2022-05-26 · Lingxiao Wang, Qi Cai, Zhuoran Yang, Zhaoran Wang

Reinforcement learning in partially observed Markov decision processes (POMDPs) faces two challenges. (i) It often takes the full history to predict the future, which induces a sample complexity that scales exponentially…

reinforcement-learning Reinforcement Learning (RL)Representation Learning

Provably Efficient Offline Reinforcement Learning in Regular Decision Processes