A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms
We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We demonstrate its effectiveness by presenting simple and unified proofs of convergence for a variety of commonly-used methods. We show that value-based methods such as TD($\lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions, thus establishing their exponentially fast convergence to a stationary distribution. We demonstrate that the stationary distribution obtained by any algorithm whose target is an expected Bellman update has a mean which is equal to the true value function. Furthermore, we establish that the distributions concentrate around their mean as the step-size shrinks. We further analyse the optimistic policy iteration algorithm, for which the contraction property does not hold, and formulate a probabilistic policy improvement property which entails the convergence of the algorithm.
Code (0)
등록된 구현이 없습니다.
Tasks
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
An Analysis of Categorical Distributional Reinforcement Learning
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performan…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorp…
Distributional Reinforcement LearningMulti-agent Reinforcement Learningquantile regressionreinforcement-learning+2Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), reso…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Accelerated Distributional Temporal Difference Learning with Linear Function Approximation
In this paper, we study the finite-sample statistical rates of distributional temporal difference (TD) learning with linear function approximation. The purpose of distributional TD learning is to estimate the return dist…
Reinforcement LearningOne-Step Distributional Reinforcement Learning
Reinforcement learning (RL) allows an agent interacting sequentially with an environment to maximize its long-term expected return. In the distributional RL (DistrRL) paradigm, the agent goes beyond the limit of the expe…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)