Identity Curvature Laplace Approximation for Improved Out-of-Distribution Detection

2023-12-16 · Maksim Zhdanov, Stanislav Dereka, Sergey Kolesnikov

Uncertainty estimation is crucial in safety-critical applications, where robust out-of-distribution (OOD) detection is essential. Traditional Bayesian methods, though effective, are often hindered by high computational demands. As an alternative, Laplace approximation offers a more practical and efficient approach to uncertainty estimation. In this paper, we introduce the Identity Curvature Laplace Approximation (ICLA), a novel method that challenges the conventional posterior covariance formulation by using identity curvature and optimizing prior precision. This innovative design significantly enhances OOD detection performance on well-known datasets such as CIFAR-10, CIFAR-100, and ImageNet, while maintaining calibration scores. We attribute this improvement to the alignment issues between typical feature embeddings and curvature as measured by the Fisher information matrix. Our findings are further supported by demonstrating that incorporating Fisher penalty or sharpness-aware minimization techniques can greatly enhance the uncertainty estimation capabilities of standard Laplace approximation.

📄 PDF Abstract BibTeX arXiv:2312.10464

Code (1)

maxnygma/icla 공식 구현 pytorch

Tasks

AttributeOut-of-Distribution Detection Out of Distribution (OOD) Detection

Methods 이 논문이 사용한 방법론

Sharpness-Aware Minimization Sharpness-Aware Minimization, or SAM, is a procedure that improves model generalization by simultaneously minimizing loss value and loss sharpness. SAM functions by…

Mitigating Forgetting in Low Rank Adaptation

2025-12-19 · Joanna Sliwa, Frank Schneider, Philipp Hennig, Jose Miguel Hernandez-Lobato arxiv

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), enable fast specialization of large pre-trained models to different downstream applications. However, this process often leads to catastrophic …

parameter-efficient fine-tuning Mathematical Reasoning

Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

2018-05-20 · NeurIPS 2018 12 · Hippolyt Ritter, Aleksandar Botev, David Barber

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate…

Permuted-MNIST

L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation

2021-07-09 · Christian S. Perone, Roberto Pereira Silveira, Thomas Paula

Uncertainty quantification for deep neural networks has recently evolved through many techniques. In this work, we revisit Laplace approximation, a classical approach for posterior approximation that is computationally a…

Uncertainty Quantification

SPDE Methods for Nonparametric Bayesian Posterior Contraction and Laplace Approximation

2026-03-23 · Enric Alberola-Boloix, Ioar Casado-Telletxea arxiv

We derive posterior contraction rates (PCRs) and finite-sample Bernstein von Mises (BvM) results for non-parametric Bayesian models by extending the diffusion-based framework of Mou et al. (2024) to the infinite-dimensio…

Variational Laplace for Bayesian neural networks

2020-11-20 · Ali Unlu, Laurence Aitchison

We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-netwo…

Benchmarking Variational Inference