A Local Temporal Difference Code for Distributional Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Pablo Tano, Peter Dayan, Alexandre Pouget

Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather than just their expected values. However, the distributional codes explored so far rely on a complex imputation step which crucially relies on spatial non-locality: in order to compute reward prediction errors, units must know not only their own state but also the states of the other units. It is far from clear how these steps could be implemented in realistic neural circuits. Here, we introduce the Laplace code: a local temporal difference code for distributional reinforcement learning that is representationally powerful and computationally straightforward. The code decomposes value distributions and prediction errors across three separated dimensions: reward magnitude (related to distributional quantiles), temporal discounting (related to the Laplace transform of future rewards) and time horizon (related to eligibility traces). Besides lending itself to a local learning rule, the decomposition recovers the temporal evolution of the immediate reward distribution, indicating all possible rewards at all future times. This increases representational capacity and allows for temporally-flexible computations that immediately adjust to changing horizons or discount factors.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement LearningImputation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Nonlinear Distributional Gradient Temporal-Difference Learning

2018-05-20 · Chao Qu, Shie Mannor, Huan Xu

We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distribu…

Distributional Reinforcement Learning Reinforcement Learning

A Finite Sample Analysis for Quantile Temporal Difference Learning in Distributional Reinforcement Learning

2026-08-27 · Zijie Cheng, Xiang Li, Yang Peng, Zhihua Zhang arxiv

We establish a global finite-sample guarantee for synchronous quantile temporal-difference learning (QTD) in tabular distributional reinforcement learning. The proof separates two stability mechanisms. A global compariso…

Reinforcement Learning

Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning

2026-08-13 · Zijie Cheng, Yang Peng, Zhihua Zhang arxiv

In this paper, we study how to perform statistical inference for quantile temporal difference learning (QTD) in distributional reinforcement learning. Assuming access to a generative model, we first establish functional …

Reinforcement Learning

The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation

2023-05-28 · Mark Rowland, Yunhao Tang, Clare Lyle, Rémi Munos 외

We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning

Distributional Bellman Operators over Mean Embeddings

2023-12-09 · Li Kevin Wenliang, Grégoire Delétang, Matthew Aitchison, Marcus Hutter 외

We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming a…

Atari Games Deep Reinforcement Learning Distributional Reinforcement Learning reinforcement-learning+1