Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates
For large nonlinear least squares loss functions in machine learning we exploit the property that the number of model parameters typically exceeds the data in one batch. This implies a low-rank structure in the Hessian of the loss, which enables effective means to compute search directions. Using this property, we develop two algorithms that estimate Jacobian matrices and perform well when compared to state-of-the-art methods.
Code (1)
Tasks
BIG-bench Machine LearningSimilar Papers 제목 키워드 기반
DeepLM: Large-Scale Nonlinear Least Squares on Deep Learning Frameworks Using Stochastic Domain Decomposition
We propose a novel approach for large-scale nonlinear least squares problems based on deep learning frameworks. Nonlinear least squares are commonly solved with the Levenberg-Marquardt (LM) algorithm for fast converg…
Deep LearningDistributed OptimizationScalable Derivative-Free Optimization for Nonlinear Least-Squares Problems
Derivative-free - or zeroth-order - optimization (DFO) has gained recent attention for its ability to solve problems in a variety of application areas, including machine learning, particularly involving objectives which …
Dimensionality ReductionDebiasing Random Oblique Projections for Subsampled OLS and Fast CUR in High Dimensions
Random sampling is a fundamental tool in modern machine learning and numerical linear algebra for reducing the computational cost of large-scale matrix problems. Existing analyses, however, rely primarily on subspace emb…
A Randomised Subspace Gauss-Newton Method for Nonlinear Least-Squares
We propose a Randomised Subspace Gauss-Newton (R-SGN) algorithm for solving nonlinear least-squares optimization problems, that uses a sketched Jacobian of the residual in the variable domain and solves a reduced linear …
regressionA rigorous introduction to linear models
This book is meant to provide an introduction to linear models and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to ordinary least squares. In machine learning, …
regression