Extending Differential Temporal Difference Methods for Episodic Problems

2026-05-06 · Kris De Asis, Mohamed Elsayed, Jiamin He arxiv

Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function's state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming reinforcement learning algorithms to their differential counterparts. Across a range of base algorithms and environments, we empirically validate that reward centering can improve sample efficiency in episodic problems.

📄 PDF Abstract BibTeX arXiv:2605.04368

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

2025-10-26 · Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini, Yash Aggarwal 외 arxiv

In-context learning is governed by both temporal and semantic relationships, shaping how Large Language Models (LLMs) retrieve contextual information. Analogous to human episodic memory, where the retrieval of specific e…

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training

2025-02-09 · Deven Mahesh Mistry, Anooshka Bajaj, Yash Aggarwal, Sahaj Singh Maini 외

We investigate in-context temporal biases in attention heads and transformer outputs. Using cognitive science methodologies, we analyze attention scores and outputs of the GPT-2 models of varying sizes. Across attention …

In-Context Learning

Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Reward

2023-12-17 · Haoxin Lin, Hongqiu Wu, Jiaji Zhang, Yihao Sun 외

Real-world decision-making problems are usually accompanied by delayed rewards, which affects the sample efficiency of Reinforcement Learning, especially in the extremely delayed case where the only feedback is the episo…

Decision Making

Temporal-Differential Learning in Continuous Environments

2020-06-01 · Tao Bian, Zhong-Ping Jiang

In this paper, a new reinforcement learning (RL) method known as the method of temporal differential is introduced. Compared to the traditional temporal-difference learning method, it plays a crucial role in developing n…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning

2025-08-14 · Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park 외 arxiv

In this work, we tackle the problem of video classincremental learning (VCIL). Many existing VCIL methods mitigate catastrophic forgetting by rehearsal training with a few temporally dense samples stored in episodic memo…

class-incremental learning General Knowledge