Offline Reinforcement Learning with Realizability and Single-policy Concentrability

2022-02-09 · Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, Jason D. Lee

Sample-efficiency guarantees for offline reinforcement learning (RL) often rely on strong assumptions on both the function classes (e.g., Bellman-completeness) and the data coverage (e.g., all-policy concentrability). Despite the recent efforts on relaxing these assumptions, existing works are only able to relax one of the two factors, leaving the strong assumption on the other factor intact. As an important open problem, can we achieve sample-efficient offline RL with weak assumptions on both factors? In this paper we answer the question in the positive. We analyze a simple algorithm based on the primal-dual formulation of MDPs, where the dual variables (discounted occupancy) are modeled using a density-ratio function against offline data. With proper regularization, we show that the algorithm enjoys polynomial sample complexity, under only realizability and single-policy concentrability. We also provide alternative analyses based on different assumptions to shed light on the nature of primal-dual algorithms for offline RL.

📄 PDF Abstract BibTeX arXiv:2202.04634

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data

2024-03-25 · Zeyu Jia, Alexander Rakhlin, Ayush Sekhari, Chen-Yu Wei

We revisit the problem of offline reinforcement learning with value function realizability but without Bellman completeness. Previous work by Xie and Jiang (2021) and Foster et al. (2022) left open the question whether a…

reinforcement-learning

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^π$-Realizability and Concentrability

2025-10-03 · Volodymyr Tkachuk, Csaba Szepesvári, Xiaoqi Tan arxiv

We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization. Prior work established that statistically efficient learning is impossible for e…

Reinforcement Learning Offline RL

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

2023-02-07 · NeurIPS 2023 11

Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks …

A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

2023-06-13 · Kihyuk Hong, Yuhang Li, Ambuj Tewari

Offline constrained reinforcement learning (RL) aims to learn a policy that maximizes the expected cumulative reward subject to constraints on expected cumulative cost using an existing dataset. In this paper, we propose…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

2021-11-21 · Dylan J. Foster, Akshay Krishnamurthy, David Simchi-Levi, Yunzong Xu

We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data. Offline RL -- particularly when coupled with (value) function approximation to allow for genera…

Decision Making Offline RL reinforcement-learning Reinforcement Learning+1