Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage

2023-02-05 · NeurIPS 2023 11

In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with $L^2$-convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.

📄 PDF Abstract BibTeX arXiv:2302.02392

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RLQ-LearningReinforcement Learning (RL)

Partial Feedback Online Learning

2026-01-29 · Shihao Shao, Cong Fang, Zhouchen Lin, Dacheng Tao arxiv

We study a new learning protocol, termed partial-feedback online learning, where each instance admits a set of acceptable labels, but the learner observes only one acceptable label per round. We highlight that, while cla…

Computational Hardness of Reinforcement Learning with Partial $q^π$-Realizability

2025-10-24 · Shayan Karimi, Xiaoqi Tan arxiv

This paper investigates the computational complexity of reinforcement learning in a novel linear function approximation regime, termed partial $q^π$-realizability. In this framework, the objective is to learn an $ε$-opti…

Reinforcement Learning

Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency

2021-02-05 · Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus 외

We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning using function approximation for marginal importance weights and $q$-functions when these are estimated using recent minima…

Off-policy evaluation reinforcement-learning Reinforcement Learning (RL)

On the Complexity of Offline Reinforcement Learning with $Q^\star$-Approximation and Partial Coverage

2026-02-12 · Haolin Liu, Braham Snyder, Chen-Yu Wei arxiv

We study offline reinforcement learning under $Q^\star$-approximation and partial coverage, a setting that motivates practical algorithms such as Conservative $Q$-Learning (CQL; Kumar et al., 2020) but has received limit…

Reinforcement Learning Offline RL

Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

2021-07-13 · ICLR 2022 4 · Masatoshi Uehara, Wen Sun

We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm named Constrained Pessimistic Policy Op…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1