L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
Uncertainty quantification for deep neural networks has recently evolved through many techniques. In this work, we revisit Laplace approximation, a classical approach for posterior approximation that is computationally attractive. However, instead of computing the curvature matrix, we show that, under some regularity conditions, the Laplace approximation can be easily constructed using the gradient second moment. This quantity is already estimated by many exponential moving average variants of Adagrad such as Adam and RMSprop, but is traditionally discarded after training. We show that our method (L2M) does not require changes in models or optimization, can be implemented in a few lines of code to yield reasonable results, and it does not require any extra computational steps besides what is already being computed by optimizers, without introducing any new hyperparameter. We hope our method can open new research directions on using quantities already computed by optimizers for uncertainty estimation in deep neural networks.
Code (0)
등록된 구현이 없습니다.
Tasks
Uncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Riemannian Laplace approximations for Bayesian neural networks
Bayesian neural networks often approximate the weight-posterior with a Gaussian distribution. However, practical posteriors are often, even locally, highly non-Gaussian, and empirical performance deteriorates. We propose…
Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
In this thesis, we disentangle the generalized Gauss-Newton and approximate inference for Bayesian deep learning. The generalized Gauss-Newton method is an optimization method that is used in several popular Bayesian dee…
Deep LearningGaussian ProcessesThe computational asymptotics of Gaussian variational inference and the Laplace approximation
Gaussian variational inference and the Laplace approximation are popular alternatives to Markov chain Monte Carlo that formulate Bayesian posterior inference as an optimization problem, enabling the use of simple and sca…
Bayesian InferenceStochastic OptimizationVariational InferenceImproving the Linearized Laplace Approximation via Quadratic Approximations
Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and…
Laplace approximation for logistic Gaussian process density estimation and regression
Logistic Gaussian process (LGP) priors provide a flexible alternative for modelling unknown densities. The smoothness properties of the density estimates can be controlled through the prior covariance structure of the LG…
Bayesian InferenceDensity Estimationregression