Properties of the Least Squares Temporal Difference learning algorithm

2013-01-22 · Kamil Ciosek

This paper presents four different ways of looking at the well-known Least Squares Temporal Differences (LSTD) algorithm for computing the value function of a Markov Reward Process, each of them leading to different insights: the operator-theory approach via the Galerkin method, the statistical approach via instrumental variables, the linear dynamical system view as well as the limit of the TD iteration. We also give a geometric view of the algorithm as an oblique projection. Furthermore, there is an extensive comparison of the optimization problem solved by LSTD as compared to Bellman Residual Minimization (BRM). We then review several schemes for the regularization of the LSTD solution. We then proceed to treat the modification of LSTD for the case of episodic Markov Reward Processes.

📄 PDF Abstract BibTeX arXiv:1301.5220

Code (0)

등록된 구현이 없습니다.

Supplementary material for Uncorrected least-squares temporal difference with lambda-return

2019-11-14 · Takayuki Osogami

Here, we provide a supplementary material for Takayuki Osogami, "Uncorrected least-squares temporal difference with lambda-return," which appears in {\it Proceedings of the 34th AAAI Conference on Artificial Intelligence…

Accelerated Gradient Temporal Difference Learning

2016-11-28 · Yangchen Pan, Adam White, Martha White

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD({\lambda}) to data efficient least squares methods. Least square methods make the best use of available da…

Concentration bounds for temporal difference learning with linear function approximation: The case of batch data and uniform sampling

2013-06-11 · L. A. Prashanth, Nathaniel Korda, Rémi Munos

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regul…

Multi-Armed Bandits News Recommendation regression Traffic Signal Control

LSTD with Random Projections

2010-12-01 · NeurIPS 2010 12 · Mohammad Ghavamzadeh, Alessandro Lazaric, Odalric Maillard, Rémi Munos

We consider the problem of reinforcement learning in high-dimensional spaces when the number of features is bigger than the number of samples. In particular, we study the least-squares temporal difference (LSTD) learning…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Randomised Bayesian Least-Squares Policy Iteration

2019-04-06 · Nikolaos Tziortziotis, Christos Dimitrakakis, Michalis Vazirgiannis

We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies…

Thompson Sampling