Adaptive Tree Backup Algorithms for Temporal-Difference Reinforcement Learning

2022-06-04 · Brett Daley, Isaac Chan

Q($\sigma$) is a recently proposed temporal-difference learning method that interpolates between learning from expected backups and sampled backups. It has been shown that intermediate values for the interpolation parameter $\sigma \in [0,1]$ perform better in practice, and therefore it is commonly believed that $\sigma$ functions as a bias-variance trade-off parameter to achieve these improvements. In our work, we disprove this notion, showing that the choice of $\sigma=0$ minimizes variance without increasing bias. This indicates that $\sigma$ must have some other effect on learning that is not fully understood. As an alternative, we hypothesize the existence of a new trade-off: larger $\sigma$-values help overcome poor initializations of the value function, at the expense of higher statistical variance. To automatically balance these considerations, we propose Adaptive Tree Backup (ATB) methods, whose weighted backups evolve as the agent gains experience. Our experiments demonstrate that adaptive strategies can be more effective than relying on fixed or time-annealed $\sigma$-values.

📄 PDF Abstract BibTeX arXiv:2206.01896

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators

2021-06-24 · NeurIPS 2021 12 · Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, Karthikeyan Shanmugam

In temporal difference (TD) learning, off-policy sampling is known to be more practical than on-policy sampling, and by decoupling learning from data collection, it enables data reuse. It is known that policy evaluation …

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

2018-02-09 · Long Yang, Minhao Shi, Qian Zheng, Wenjia Meng 외

Recently, a new multi-step temporal learning algorithm, called $Q(\sigma)$, unifies $n$-step Tree-Backup (when $\sigma=0$) and $n$-step Sarsa (when $\sigma=1$) by introducing a sampling parameter $\sigma$. However, simil…

Reinforcement Learning

TD_gamma: Re-evaluating Complex Backups in Temporal Difference Learning

2011-12-01 · NeurIPS 2011 12 · George Konidaris, Scott Niekum, Philip S. Thomas

We show that the lambda-return target used in the TD(lambda) family of algorithms is the maximum likelihood estimator for a specific model of how the variance of an n-step return estimate increases with n. We introduce t…

Convergent Tree Backup and Retrace with Function Approximation

2017-05-25 · ICML 2018 7 · Ahmed Touati, Pierre-Luc Bacon, Doina Precup, Pascal Vincent

Off-policy learning is key to scaling up reinforcement learning as it allows to learn about a target policy from the experience generated by a different behavior policy. Unfortunately, it has been challenging to combine …

Reinforcement Learning

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

2019-01-22 · J. Fernando Hernandez-Garcia, Richard S. Sutton

Multi-step methods such as Retrace($\lambda$) and $n$-step $Q$-learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures an…

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1