One-Step Distributional Reinforcement Learning

2023-04-27 · Mastane Achab, REDA ALAMI, Yasser Abdelaziz Dahou Djilali, Kirill Fedyanin, Eric Moulines

Reinforcement learning (RL) allows an agent interacting sequentially with an environment to maximize its long-term expected return. In the distributional RL (DistrRL) paradigm, the agent goes beyond the limit of the expected value, to capture the underlying probability distribution of the return across all time steps. The set of DistrRL algorithms has led to improved empirical performance. Nevertheless, the theory of DistrRL is still not fully understood, especially in the control case. In this paper, we present the simpler one-step distributional reinforcement learning (OS-DistrRL) framework encompassing only the randomness induced by the one-step dynamics of the environment. Contrary to DistrRL, we show that our approach comes with a unified theory for both policy evaluation and control. Indeed, we propose two OS-DistrRL algorithms for which we provide an almost sure convergence analysis. The proposed approach compares favorably with categorical DistrRL on various environments.

📄 PDF Abstract BibTeX arXiv:2304.14421

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning

2022-07-15 · Yunhao Tang, Mark Rowland, Rémi Munos, Bernardo Ávila Pires 외

We study the multi-step off-policy learning approach to distributional RL. Despite the apparent similarity between value-based RL and distributional RL, our study reveals intriguing and fundamental differences between th…

Distributional Reinforcement Learning quantile regression reinforcement-learning Reinforcement Learning (RL)

Distributional Reinforcement Learning with Monotonic Splines

2021-09-29 · ICLR 2022 4 · Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte 외

Distributional Reinforcement Learning (RL) differs from traditional RL by estimating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to param…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Federated Distributional Reinforcement Learning with Distributional Critic Regularization

2026-03-18 · David Millard, Cecilia Alm, Rashid Ali, Pengcheng Shi 외 arxiv

Federated reinforcement learning typically aggregates value functions or policies by parameter averaging, which emphasizes expected return and can obscure statistical multimodality and tail behavior that matter in safety…

Reinforcement Learning

A Local Temporal Difference Code for Distributional Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Pablo Tano, Peter Dayan, Alexandre Pouget

Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather th…

Distributional Reinforcement Learning Imputation reinforcement-learning Reinforcement Learning+1

2048: Reinforcement Learning in a Delayed Reward Environment

2025-07-07 · Prady Saligram, Tanvir Bhathal, Robby Manihani

Delayed and sparse rewards present a fundamental obstacle for reinforcement-learning (RL) agents, which struggle to assign credit for actions whose benefits emerge many steps later. The sliding-tile game 2048 epitomizes …

quantile regression reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)