Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

2021-11-19 · Hao Chen, Lili Zheng, Raed Al Kontar, Garvesh Raskutti

Stochastic gradient descent (SGD) and its variants have established themselves as the go-to algorithms for large-scale machine learning problems with independent samples due to their generalization performance and intrinsic computational advantage. However, the fact that the stochastic gradient is a biased estimator of the full gradient with correlated samples has led to the lack of theoretical understanding of how SGD behaves under correlated settings and hindered its use in such cases. In this paper, we focus on hyperparameter estimation for the Gaussian process (GP) and take a step forward towards breaking the barrier by proving minibatch SGD converges to a critical point of the full log-likelihood loss function, and recovers model hyperparameters with rate $O(\frac{1}{K})$ for $K$ iterations, up to a statistical error term depending on the minibatch size. Our theoretical guarantees hold provided that the kernel functions exhibit exponential or polynomial eigendecay which is satisfied by a wide range of kernels commonly used in GPs. Numerical studies on both simulated and real datasets demonstrate that minibatch SGD has better generalization over state-of-the-art GP methods while reducing the computational burden and opening a new, previously unexplored, data size regime for GPs.

📄 PDF Abstract BibTeX arXiv:2111.10461

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Amortized variance reduction for doubly stochastic objectives

2020-03-09 · Ayman Boustati, Sattar Vakili, James Hensman, ST John

Approximate inference in complex probabilistic models such as deep Gaussian processes requires the optimisation of doubly stochastic objective functions. These objectives incorporate randomness both from mini-batch subsa…

Gaussian Processes

Scalable Training of Inference Networks for Gaussian-Process Models

2019-05-27 · Jiaxin Shi, Mohammad Emtiyaz Khan, Jun Zhu

Inference in Gaussian process (GP) models is computationally challenging for large data, and often difficult to approximate with a small number of inducing points. We explore an alternative approximation that employs sto…

Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inference

2022-02-25 · Vidhi Lalchand, Aditya Ravuri, Neil D. Lawrence

Gaussian process latent variable models (GPLVM) are a flexible and non-linear approach to dimensionality reduction, extending classical Gaussian processes to an unsupervised learning context. The Bayesian incarnation of …

Benchmarking Dimensionality Reduction Gaussian Processes Variational Inference

Gaussian Process Latent Variable Flows for Massively Missing Data

2020-11-23 · pproximateinference AABI Symposium 2021 1 · Vidhi Lalchand, Aditya Ravuri, Neil D Lawrence

Gaussian process latent variable models (GPLVM) are used to perform nonlinear and probabilistic dimensionality reduction. They extend Gaussian processes (GP) to the domain of unsupervised learning. The Bayesian incarnati…

Dimensionality Reduction Gaussian Processes Normalising Flows Variational Inference

Scalable Multi-Output Gaussian Processes with Stochastic Variational Inference

2024-07-02 · Xiaoyu Jiang, Sokratia Georgaka, Magnus Rattray, Mauricio A. Alvarez

The Multi-Output Gaussian Process is is a popular tool for modelling data from multiple sources. A typical choice to build a covariance function for a MOGP is the Linear Model of Coregionalization (LMC) which parametrica…

Gaussian Processes Variational Inference