A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

2026-06-16 · M. Forzo, E. Monzio Compagnoni, A. Russo, A. Pacchiano arxiv

Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.

📄 PDF Abstract BibTeX arXiv:2606.18183

Code (0)

등록된 구현이 없습니다.

Kalman Temporal Differences

2014-01-16 · Matthieu Geist, Olivier Pietquin

Because reinforcement learning suffers from a lack of scalability, online value (and Q-) function approximation has received increasing interest this last decade. This contribution introduces a novel approximation scheme…

Management reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Two-Timescale Networks for Nonlinear Value Function Approximation

2019-05-01 · ICLR 2019 5 · Wesley Chung, Somjit Nath, Ajin Joseph, Martha White

A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approxim…

Q-Learning Reinforcement Learning Vocal Bursts Valence Prediction

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

2024-09-18 · Jiuqi Wang, Shangtong Zhang

Temporal difference (TD) learning with linear function approximation, abbreviated as linear TD, is a classic and powerful prediction algorithm in reinforcement learning. While it is well understood that linear TD converg…

Predictive State Temporal Difference Learning

2010-12-01 · NeurIPS 2010 12 · Byron Boots, Geoffrey J. Gordon

We propose a new approach to value function approximation which combines linear temporal difference reinforcement learning with subspace identification. In practical applications, reinforcement learning (RL) is complicat…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

$\ell_1$ Regularized Gradient Temporal-Difference Learning

2016-10-05 · Dominik Meyer, Hao Shen, Klaus Diepold

In this paper, we study the Temporal Difference (TD) learning with linear value function approximation. It is well known that most TD learning algorithms are unstable with linear function approximation and off-policy lea…