Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient

2022-10-03 · Ming Yin, Mengdi Wang, Yu-Xiang Wang

Offline reinforcement learning, which aims at optimizing sequential decision-making strategies with historical data, has been extensively applied in real-life applications. State-Of-The-Art algorithms usually leverage powerful function approximators (e.g. neural networks) to alleviate the sample complexity hurdle for better empirical performances. Despite the successes, a more systematic understanding of the statistical complexity for function approximation remains lacking. Towards bridging the gap, we take a step by considering offline reinforcement learning with differentiable function class approximation (DFA). This function class naturally incorporates a wide range of models with nonlinear/nonconvex structures. Most importantly, we show offline RL with differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning (PFQL) algorithm, and our results provide the theoretical basis for understanding a variety of practical heuristics that rely on Fitted Q-Iteration style design. In addition, we further improve our guarantee with a tighter instance-dependent characterization. We hope our work could draw interest in studying reinforcement learning with differentiable function approximation beyond the scope of current research.

📄 PDF Abstract BibTeX arXiv:2210.00750

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making Offline RLQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

2026-04-15 · Shangzhe Li, Weitong Zhang arxiv

We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$-function, the learner aims to adapt it to the target environme…

Reinforcement Learning

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

2023-10-02 · Qiwei Di, Heyang Zhao, Jiafan He, Quanquan Gu

Offline reinforcement learning (RL), where the agent aims to learn the optimal policy based on the data collected by a behavior policy, has attracted increasing attention in recent years. While offline RL with linear fun…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

2023-02-07 · NeurIPS 2023 11

Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks …

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

2022-05-26 · Miao Lu, Yifei Min, Zhaoran Wang, Zhuoran Yang

We study offline reinforcement learning (RL) in partially observable Markov decision processes. In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the…

Causal Inference Offline RL Reinforcement Learning (RL)

Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration

2021-12-01 · NeurIPS 2021 12 · Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang

In constrained multi-objective RL, the goal is to learn a policy that achieves the best performance specified by a multi-objective preference function under a constraint. We focus on the offline setting where the RL agen…

Multi-Objective Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)