TD_gamma: Re-evaluating Complex Backups in Temporal Difference Learning

2011-12-01 · NeurIPS 2011 12 · George Konidaris, Scott Niekum, Philip S. Thomas

We show that the lambda-return target used in the TD(lambda) family of algorithms is the maximum likelihood estimator for a specific model of how the variance of an n-step return estimate increases with n. We introduce the gamma-return estimator, an alternative target based on a more accurate model of variance, which defines the TD_gamma family of complex-backup temporal difference learning algorithms. We derive TD_gamma, the gamma-return equivalent of the original TD(lambda) algorithm, which eliminates the lambda parameter but can only perform updates at the end of an episode and requires time and space proportional to the episode length. We then derive a second algorithm, TD_gamma(C), with a capacity parameter C. TD_gamma(C) requires C times more time and memory than TD(lambda) and is incremental and online. We show that TD_gamma outperforms TD(lambda) for any setting of lambda on 4 out of 5 benchmark domains, and that TD_gamma(C) performs as well as or better than TD_gamma for intermediate settings of C.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Adaptive Tree Backup Algorithms for Temporal-Difference Reinforcement Learning

2022-06-04 · Brett Daley, Isaac Chan

Q($\sigma$) is a recently proposed temporal-difference learning method that interpolates between learning from expected backups and sampled backups. It has been shown that intermediate values for the interpolation parame…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Temporal Difference Learning as Gradient Splitting

2020-10-27 · Rui Liu, Alex Olshevsky

Temporal difference learning with linear function approximation is a popular method to obtain a low-dimensional approximation of the value function of a policy in a Markov Decision Process. We give a new interpretation o…

A Local Temporal Difference Code for Distributional Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Pablo Tano, Peter Dayan, Alexandre Pouget

Recent theoretical and experimental results suggest that the dopamine system implements distributional temporal difference backups, allowing learning of the entire distributions of the long-run values of states rather th…

Distributional Reinforcement Learning Imputation reinforcement-learning Reinforcement Learning+1

Horizon Adaptive Offline Policy Learning via Value Stitching

2026-06-19 · Kexin Zheng, Xianyuan Zhan, Xintao Yan arxiv

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation …

Reinforcement Learning Offline RL

Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces

2024-03-09 · Yang Peng, Liangyu Zhang, Zhihua Zhang

Distributional reinforcement learning (DRL) has achieved empirical success in various domains. One core task in DRL is distributional policy evaluation, which involves estimating the return distribution $\eta^\pi$ for a …

Distributional Reinforcement Learning