Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems
Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure $\mu$) using Tikhonov least squares regression against a set of targets, will approximate the target function in the $L^2(\mu)$ norm. In the special case that the targets are future observations, the ESN is learning the next step map, which allows time series forecasting. We demonstrate the theory numerically by training an ESN using Tikhonov least squares on a sequence of scalar observations of the Lorenz system.
Code (0)
등록된 구현이 없습니다.
Tasks
regressionTime SeriesTime Series AnalysisTime Series ForecastingSimilar Papers 제목 키워드 기반
Variance reduction in stochastic methods for large-scale regularised least-squares problems
Large dimensional least-squares and regularised least-squares problems are expensive to solve. There exist many approximate techniques, some deterministic (like conjugate gradient), some stochastic (like stochastic gradi…
Point ProcessesComparison of echo state network output layer classification methods on noisy data
Echo state networks are a recently developed type of recurrent neural network where the internal layer is fixed with random weights, and only the output layer is trained on specific data. Echo state networks are increasi…
Event DetectionGeneral Classificationspeech-recognitionSpeech RecognitionBeyond Tikhonov: Faster Learning with Self-Concordant Losses via Iterative Regularization
The theory of spectral filtering is a remarkable tool to understand the statistical properties of learning with kernels. For least squares, it allows to derive various regularization schemes that yield faster convergence…
Beyond Tikhonov: faster learning with self-concordant losses, via iterative regularization
The theory of spectral filtering is a remarkable tool to understand the statistical properties of learning with kernels. For least squares, it allows to derive various regularization schemes that yield faster convergence…
Recursive Least Squares Policy Control with Echo State Network
The echo state network (ESN) is a special type of recurrent neural networks for processing the time-series dataset. However, limited by the strong correlation among sequential samples of the agent, ESN-based policy contr…
Time SeriesTime Series Analysis