Expected Eligibility Traces

2020-07-03 · Hado van Hasselt, Sephora Madjiheurem, Matteo Hessel, David Silver, André Barreto, Diana Borsa

The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that could also have led to the current state. In this work, we introduce expected eligibility traces. Expected traces allow, with a single update, to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We discuss when expected traces provide benefits over classic (instantaneous) traces in temporal-difference learning, and show that sometimes substantial improvements can be attained. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping, which ensures that the resulting algorithm is a strict generalisation of TD($\lambda$). Finally, we discuss possible extensions and connections to related ideas, such as successor features.

📄 PDF Abstract BibTeX arXiv:2007.01839

Code (0)

등록된 구현이 없습니다.

Tasks

counterfactual

From Past to Future: Rethinking Eligibility Traces

2023-12-20 · Dhawal Gupta, Scott M. Jordan, Shreyas Chaudhari, Bo Liu 외

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in…

Investigating Recurrence and Eligibility Traces in Deep Q-Networks

2017-04-18 · Jean Harb, Doina Precup

Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowledge back over time-steps in a single update. We investigate the use of eligibil…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning

2020-08-23 · Taisuke Kobayashi

Deep reinforcement learning (DRL) is one promising approach to teaching robots to perform complex tasks. Because methods that directly reuse the stored experience data cannot follow the change of the environment in robot…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Learning From the Past with Cascading Eligibility Traces

2025-06-17 · Tokiniaina Raharison Ralambomihanta, Ivan Anokhin, Roman Pogodin, Samira Ebrahimi Kahou 외

Animals often receive information about errors and rewards after a significant delay. For example, there is typically a delay of tens to hundreds of milliseconds between motor actions and visual feedback. The standard ap…

Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR

2026-05-07 · Chaoli Mou, Zhan Zhuang, Xinning Chen, Yu Zhang arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Op…

Reinforcement Learning