Distributional reinforcement learning with linear function approximation

2019-02-08 · Marc G. Bellemare, Nicolas Le Roux, Pablo Samuel Castro, Subhodeep Moitra

Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited. One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cram\'er distance, but their results only apply to the tabular setting and ignore C51's use of a softmax to produce normalized distributions. In this paper we adapt the Cram\'er distance to deal with arbitrary vectors. From it we derive a new distributional algorithm which is fully Cram\'er-based and can be combined to linear function approximation, with formal guarantees in the context of policy evaluation. In allowing the model's prediction to be any real vector, we lose the probabilistic interpretation behind the method, but otherwise maintain the appealing properties of distributional approaches. To the best of our knowledge, ours is the first proof of convergence of a distributional algorithm combined with function approximation. Perhaps surprisingly, our results provide evidence that Cram\'er-based distributional methods may perform worse than directly approximating the value function.

📄 PDF Abstract BibTeX arXiv:1902.03149

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation

2025-02-20 · Yang Peng, Kaicheng Jin, Liangyu Zhang, Zhihua Zhang

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The aim of distributional TD learning is to estimate the return distribu…

Distributional Reinforcement Learning

Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

2025-11-16 · Kaicheng Jin, Yang Peng, Jiansheng Yang, Zhihua Zhang arxiv

In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return dist…

Reinforcement Learning

Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation

2024-02-23 · Zhishuai Liu, Pan Xu

We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision …

Reinforcement Learning (RL)

A Comparative Analysis of Expected and Distributional Reinforcement Learning

2019-01-30 · Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). Howe…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

2022-09-14 · Xiaoteng Ma, Zhipeng Liang, Jose Blanchet, Mingwen Liu 외

Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy i…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)