On Policy Evaluation Algorithms in Distributional Reinforcement Learning

2024-07-19 · Julian Gerstenberg, Ralph Neininger, Denis Spiegel

We introduce a novel class of algorithms to efficiently approximate the unknown return distributions in policy evaluation problems from distributional reinforcement learning (DRL). The proposed distributional dynamic programming algorithms are suitable for underlying Markov decision processes (MDPs) having an arbitrary probabilistic reward mechanism, including continuous reward distributions with unbounded support being potentially heavy-tailed. For a plain instance of our proposed class of algorithms we prove error bounds, both within Wasserstein and Kolmogorov--Smirnov distances. Furthermore, for return distributions having probability density functions the algorithms yield approximations for these densities; error bounds are given within supremum norm. We introduce the concept of quantile-spline discretizations to come up with algorithms showing promising results in simulation experiments. While the performance of our algorithms can rigorously be analysed they can be seen as universal black box algorithms applicable to a large class of MDPs. We also derive new properties of probability metrics commonly used in DRL on which our quantitative analysis is based.

📄 PDF Abstract BibTeX arXiv:2407.14175

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement Learningreinforcement-learning Reinforcement Learning

The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning

2017-04-15 · ICLR 2018 1 · Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot 외

In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang e…

Atari Games Distributional Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

One-Step Distributional Reinforcement Learning

2023-04-27 · Mastane Achab, REDA ALAMI, Yasser Abdelaziz Dahou Djilali, Kirill Fedyanin 외

Reinforcement learning (RL) allows an agent interacting sequentially with an environment to maximize its long-term expected return. In the distributional RL (DistrRL) paradigm, the agent goes beyond the limit of the expe…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Nonlinear Distributional Gradient Temporal-Difference Learning

2018-05-20 · Chao Qu, Shie Mannor, Huan Xu

We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distribu…

Distributional Reinforcement Learning Reinforcement Learning

A Principled Path to Fitted Distributional Evaluation

2025-06-24 · Sungee Hong, Jiayi Wang, Zhengling Qi, Raymond Ka Wai Wong

In reinforcement learning, distributional off-policy evaluation (OPE) focuses on estimating the return distribution of a target policy using offline data collected under a different policy. This work focuses on extending…

Atari Games Off-policy evaluation reinforcement-learning Reinforcement Learning

Implicit Distributional Reinforcement Learning

2020-07-13 · NeurIPS 2020 12 · Yuguang Yue, Zhendong Wang, Mingyuan Zhou

To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor-critic (IDAC) that consists of a distributional critic, built on two deep generator ne…

Distributional Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning+1