Temporal Difference Learning as Gradient Splitting
Temporal difference learning with linear function approximation is a popular method to obtain a low-dimensional approximation of the value function of a policy in a Markov Decision Process. We give a new interpretation of this method in terms of a splitting of the gradient of an appropriately chosen function. As a consequence of this interpretation, convergence proofs for gradient descent can be applied almost verbatim to temporal difference learning. Beyond giving a new, fuller explanation of why temporal difference works, our interpretation also yields improved convergence times. We consider the setting with $1/\sqrt{T}$ step-size, where previous comparable finite-time convergence time bounds for temporal difference learning had the multiplicative factor $1/(1-\gamma)$ in front of the bound, with $\gamma$ being the discount factor. We show that a minor variation on TD learning which estimates the mean of the value function separately has a convergence time where $1/(1-\gamma)$ only multiplies an asymptotically negligible term.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Closing the gap between SVRG and TD-SVRG with Gradient Splitting
Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have sought to fuse TD learning with Stochastic…
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces
In this paper, we set forth a new vision of reinforcement learning developed by us over the past few years, one that yields mathematically rigorous solutions to longstanding important questions that have remained unresol…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2On Convergent Finite Difference Schemes for Variational - PDE Based Image Processing
We study an adaptive anisotropic Huber functional based image restoration scheme. By using a combination of L2-L1 regularization functions, an adaptive Huber functional based energy minimization model provides denoising …
DenoisingImage DenoisingImage RestorationA Multidimensional Exponential Utility Indifference Pricing Model with Applications to Counterparty Risk
This paper considers exponential utility indifference pricing for a multidimensional non-traded assets model subject to inter-temporal default risk, and provides a semigroup approximation for the utility indifference pri…
Gradient descent temporal difference-difference learning
Off-policy learning algorithms, in which an agent updates the value function of the optimal policy while selecting actions using an independent exploration policy, provide an effective solution to the explore-exploit tra…