Subspace Inference for Bayesian Deep Learning
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well calibrated predictive uncertainty for both regression and image classification.
Code (1)
Tasks
Bayesian InferenceDeep Learningimage-classificationImage ClassificationregressionVariational InferenceSimilar Papers 제목 키워드 기반
Learning Active Subspaces for Effective and Scalable Uncertainty Quantification in Deep Neural Networks
Bayesian inference for neural networks, or Bayesian deep learning, has the potential to provide well-calibrated predictions with quantified uncertainty and robustness. However, the main hurdle for Bayesian deep learning …
Bayesian InferenceDeep LearningUncertainty QuantificationVariational InferenceScalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference
Despite their widespread use, large language models (LLMs) are known to hallucinate incorrect information and be poorly calibrated. This makes the uncertainty quantification of these models of critical importance, especi…
Bayesian InferenceUncertainty QuantificationVariational InferenceOptimal Subspace Inference for the Laplace Approximation of Bayesian Neural Networks
Subspace inference for neural networks assumes that a subspace of their parameter space suffices to produce a reliable uncertainty quantification. In this work, we mathematically derive the optimal subspace model to a Ba…
Bayesian InferenceUncertainty QuantificationTraining Bayesian Neural Networks with Sparse Subspace Variational Inference
Bayesian neural networks (BNNs) offer uncertainty quantification but come with the downside of substantially increased training and inference costs. Sparse BNNs have been investigated for efficient inference, typically b…
Uncertainty QuantificationVariational InferenceBayesian Semi-structured Subspace Inference
Semi-structured regression models enable the joint modeling of interpretable structured and complex unstructured feature effects. The structured model part is inspired by statistical models and can be used to infer the i…
regression