Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees
Gaussian processes are frequently deployed as part of larger machine learning and decision-making systems, for instance in geospatial modeling, Bayesian optimization, or in latent Gaussian models. Within a system, the Gaussian process model needs to perform in a stable and reliable manner to ensure it interacts correctly with other parts of the system. In this work, we study the numerical stability of scalable sparse approximations based on inducing points. To do so, we first review numerical stability, and illustrate typical situations in which Gaussian process models can be unstable. Building on stability theory originally developed in the interpolation literature, we derive sufficient and in certain cases necessary conditions on the inducing points for the computations performed to be numerically stable. For low-dimensional tasks such as geospatial modeling, we propose an automated method for computing inducing points satisfying these conditions. This is done via a modification of the cover tree data structure, which is of independent interest. We additionally propose an alternative sparse approximation for regression with a Gaussian likelihood which trades off a small amount of performance to further improve stability. We provide illustrative examples showing the relationship between stability of calculations and predictive performance of inducing point methods on spatial tasks.
Code (1)
Tasks
Bayesian OptimizationDecision MakingGaussian ProcessesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improved Inverse-Free Variational Bounds for Sparse Gaussian Processes
The need for matrix decompositions (inverses) is often named as a major impediment to scaling Gaussian process (GP) models, even in efficient approximations. To address this, Van der Wilk et al. (2020) introduced a varia…
Gaussian ProcessesKernel Multigrid: Accelerate Back-fitting via Sparse Gaussian Process Regression
Additive Gaussian Processes (GPs) are popular approaches for nonparametric feature selection. The common training method for these models is Bayesian Back-fitting. However, the convergence rate of Back-fitting in trainin…
feature selectionGaussian ProcessesGPRregressionStringNET: Neural Network based Variational Method for Transition Pathways
Rare transition events in meta-stable systems under noisy fluctuations are crucial for many non-equilibrium physical and chemical processes. In these processes, the primary contributions to reactive flux are predominantl…
ARCComputational chemistryFully Scalable Gaussian Processes using Subspace Inducing Inputs
We introduce fully scalable Gaussian processes, an implementation scheme that tackles the problem of treating a high number of training instances together with high dimensional input data. Our key idea is a representatio…
Extreme Multi-Label ClassificationGaussian ProcessesGeneral ClassificationMulti-Label Classification+1Stable Recovery Of Sparse Vectors From Random Sinusoidal Feature Maps
Random sinusoidal features are a popular approach for speeding up kernel-based inference in large datasets. Prior to the inference stage, the approach suggests performing dimensionality reduction by first multiplying eac…
Dimensionality Reduction