Scale invariant process regression: Towards Bayesian ML with minimal assumptions
Current methods for regularization in machine learning require quite specific model assumptions (e.g. a kernel shape) that are not derived from prior knowledge about the application, but must be imposed merely to make the method work. We show in this paper that regularization can indeed be achieved by assuming nothing but invariance principles (w.r.t. scaling, translation, and rotation of input and output space) and the degree of differentiability of the true function. Concretely, we derive a novel (non-Gaussian) stochastic process from the above minimal assumptions, and we present a corresponding Bayesian inference method for regression. The mean posterior turns out to be a polyharmonic spline, and the posterior process is a mixture of t-processes. Compared with Gaussian process regression, the proposed method shows equal performance and has the advantages of being (i) less arbitrary (no choice of kernel) (ii) potentially faster (no kernel parameter optimization), and (iii) having better extrapolation behavior. We believe that the proposed theory has central importance for the conceptual foundations of regularization and machine learning and also has great potential to enable practical advances in ML areas beyond regression.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceGaussian ProcessesregressionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Invariant polynomials and machine learning
We present an application of invariant polynomials in machine learning. Using the methods developed in previous work, we obtain two types of generators of the Lorentz- and permutation-invariant polynomials in particle mo…
Bayesian InferenceBIG-bench Machine LearningBayesian Approximate Kernel Regression with Variable Selection
Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlik…
Binary ClassificationregressionVariable SelectionFast and Order-invariant Inference in Bayesian VARs with Non-Parametric Shocks
The shocks which hit macroeconomic models such as Vector Autoregressions (VARs) have the potential to be non-Gaussian, exhibiting asymmetries and fat tails. This consideration motivates the VAR developed in this paper wh…
Bayesian InferenceWavelet Scattering Regression of Quantum Chemical Energies
We introduce multiscale invariant dictionaries to estimate quantum chemical energies of organic molecules, from training databases. Molecular energies are invariant to isometric atomic displacements, and are Lipschitz co…
regressionCanonical Bayesian Linear System Identification
Standard Bayesian approaches for linear time-invariant (LTI) system identification are hindered by parameter non-identifiability; the resulting complex, multi-modal posteriors make inference inefficient and impractical. …
Computational Efficiency