Nonlinear Distributional Gradient Temporal-Difference Learning

2018-05-20 · Chao Qu, Shie Mannor, Huan Xu

We devise a distributional variant of gradient temporal-difference (TD) learning. Distributional reinforcement learning has been demonstrated to outperform the regular one in the recent study \citep{bellemare2017distributional}. In the policy evaluation setting, we design two new algorithms called distributional GTD2 and distributional TDC using the Cram{\'e}r distance on the distributional version of the Bellman error objective function, which inherits advantages of both the nonlinear gradient TD algorithms and the distributional RL approach. In the control setting, we propose the distributional Greedy-GQ using the similar derivation. We prove the asymptotic almost-sure convergence of distributional GTD2 and TDC to a local optimal solution for general smooth function approximators, which includes neural networks that have been widely used in recent study to solve the real-life RL problems. In each step, the computational complexities of above three algorithms are linear w.r.t.\ the number of the parameters of the function approximator, thus can be implemented efficiently for neural networks.

📄 PDF Abstract BibTeX arXiv:1805.07732

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement LearningReinforcement Learning

Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation

2019-11-11 · ICML 2020 1 · Shangtong Zhang, Bo Liu, Hengshuai Yao, Shimon Whiteson

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained vi…

Vocal Bursts Valence Prediction

A Local Temporal Difference Code for Distributional Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Pablo Tano, Peter Dayan, Alexandre Pouget

Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather th…

Distributional Reinforcement Learning Imputation reinforcement-learning Reinforcement Learning+1

Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation

2009-12-01 · NeurIPS 2009 12 · Shalabh Bhatnagar, Doina Precup, David Silver, Richard S. Sutton 외

We introduce the first temporal-difference learning algorithms that converge with smooth value function approximators, such as neural networks. Conventional temporal-difference (TD) methods, such as TD($\lambda$), Q-l…

Q-Learning

Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policies

2018-05-02 · Yann Ollivier

In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not av…

Reinforcement Learning

Two-Timescale Networks for Nonlinear Value Function Approximation

2019-05-01 · ICLR 2019 5 · Wesley Chung, Somjit Nath, Ajin Joseph, Martha White

A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approxim…

Q-Learning Reinforcement Learning Vocal Bursts Valence Prediction