Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

2025-11-16 · Kaicheng Jin, Yang Peng, Jiansheng Yang, Zhihua Zhang arxiv

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return distribution of a discounted Markov decision process for a given policy. Previous works on statistical analysis of distributional TD learning focus mainly on the tabular case. We first consider the linear function approximation setting and conduct a fine-grained analysis of the linear-categorical Bellman equation. Building on this analysis, we further incorporate variance reduction techniques in our new algorithms to establish tight sample complexity bounds independent of the support size $K$ when $K$ is large. Our theoretical results imply that, when employing distributional TD learning with linear function approximation, learning the full distribution of the return function from streaming data is no more difficult than learning its expectation. This work provide new insights into the statistical efficiency of distributional reinforcement learning algorithms.

📄 PDF Abstract BibTeX arXiv:2511.12688

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Nonlinear Distributional Gradient Temporal-Difference Learning

2018-05-20 · Chao Qu, Shie Mannor, Huan Xu

We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distribu…

Distributional Reinforcement Learning Reinforcement Learning

A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation

2025-02-20 · Yang Peng, Kaicheng Jin, Liangyu Zhang, Zhihua Zhang

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The aim of distributional TD learning is to estimate the return distribu…

Distributional Reinforcement Learning

Distributional Reinforcement Learning with Monotonic Splines

2021-09-29 · ICLR 2022 4 · Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte 외

Distributional Reinforcement Learning (RL) differs from traditional RL by estimating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to param…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning with Polynomial Sample Complexity

2020-06-06 · Bo Liu, Ian Gemp, Mohammad Ghavamzadeh, Ji Liu 외

In this paper, we introduce proximal gradient temporal difference learning, which provides a principled way of designing and analyzing true stochastic gradient temporal difference learning algorithms. We show how gradien…

reinforcement-learning Reinforcement Learning (RL)

Accelerated and instance-optimal policy evaluation with linear function approximation

2021-12-24 · Tianjiao Li, Guanghui Lan, Ashwin Pananjady

We study the problem of policy evaluation with linear function approximation and present efficient and practical algorithms that come with strong optimality guarantees. We begin by proving lower bounds that establish bas…