Provable Offline Reinforcement Learning for Structured Cyclic MDPs

2026-02-12 · Kyungbok Lee, Angelica Cristello Sarteau, Michael R. Kosorok arxiv

We introduce a novel cyclic Markov decision process (MDP) framework for multi-step decision problems with heterogeneous stage-specific dynamics, transitions, and discount factors across the cycle. In this setting, offline learning is challenging: optimizing a policy at any stage shifts the state distributions of subsequent stages, propagating mismatch across the cycle. To address this, we propose a modular structural framework that decomposes the cyclic process into stage-wise sub-problems. While generally applicable, we instantiate this principle as CycleFQI, an extension of fitted Q-iteration enabling theoretical analysis and interpretation. It uses a vector of stage-specific Q-functions, tailored to each stage, to capture within-stage sequences and transitions between stages. This modular design enables partial control, allowing some stages to be optimized while others follow predefined policies. We establish finite-sample suboptimality error bounds and derive global convergence rates under Besov regularity, demonstrating that CycleFQI mitigates the curse of dimensionality compared to monolithic baselines. Additionally, we propose a sieve-based method for asymptotic inference of optimal policy values under a margin condition. Experiments on simulated and real-world Type 1 Diabetes data sets demonstrate CycleFQI's effectiveness.

📄 PDF Abstract BibTeX arXiv:2602.11679

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning

2023-02-27 · Hao Hu, Yiqin Yang, Qianchuan Zhao, Chongjie Zhang

Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcemen…

Offline RL reinforcement-learning Reinforcement Learning (RL)

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

2022-02-23 · ICLR 2022 4 · Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng 외

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolatio…

D4RL Offline RL reinforcement-learning Reinforcement Learning+2

Leveraging Offline Data in Online Reinforcement Learning

2022-11-09 · Andrew Wagenmaker, Aldo Pacchiano

Two central paradigms have emerged in the reinforcement learning (RL) community: online RL and offline RL. In the online RL setting, the agent has no prior knowledge of the environment, and must interact with it in order…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning

2021-11-15 · Vincent Liu, James R. Wright, Martha White

Offline reinforcement learning -- learning a policy from a batch of data -- is known to be hard for general MDPs. These results motivate the need to look at specific classes of MDPs where offline reinforcement learning m…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning Foundation Models Should Already Be A Thing

2026-06-17 · Abdelrahman Zighem, Jill-Jênn Vie arxiv

Foundation models for language and vision are powered by internet-scale data, while structured domains such as tabular prediction are powered by synthetic data. This substitute shifts the challenge from collection to pri…