Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning
Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.
Code (1)
Tasks
reinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Polynomial Tensor Sketch for Element-wise Function of Low-Rank Matrix
This paper studies how to sketch element-wise functions of low-rank matrices. Formally, given low-rank matrix A = [Aij] and scalar non-linear function f, we aim for finding an approximated low-rank representation of the …
regressionNonnegative Low Rank Tensor Approximation and its Application to Multi-dimensional Images
The main aim of this paper is to develop a new algorithm for computing nonnegative low rank tensor approximation for nonnegative tensors that arise in many multi-dimensional imaging applications. Nonnegativity is one of …
Effective Streaming Low-tubal-rank Tensor Approximation via Frequent Directions
Low-tubal-rank tensor approximation has been proposed to analyze large-scale and multi-dimensional data. However, finding such an accurate approximation is challenging in the streaming setting, due to the limited computa…
Low-Rank Tensor Completion: A Pseudo-Bayesian Learning Approach
Low rank tensor completion, which solves a linear inverse problem with the principle of parsimony, is a powerful technique used in many application domains in computer vision and pattern recognition. As a surrogate funct…
Aspects of density approximation by tensor trains
Point-mass filters solve Bayesian recursive relations by approximating probability density functions of a system state over grids of discrete points. The approach suffers from the curse of dimensionality. The exponential…
State Estimation