On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors
Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Problem of the Priors, or Posteriors?
The problem of the priors is well known: it concerns the challenge of identifying norms that govern one's prior credences. I argue that a key to addressing this problem lies in considering what I call the problem of the …
PhilosophyMonte Carlo guided Diffusion for Bayesian linear inverse problems
Ill-posed linear inverse problems arise frequently in various applications, from computational photography to medical imaging. A recent line of research exploits Bayesian inference with informative priors to handle the i…
Bayesian InferenceGaussian processes for Bayesian inverse problems associated with linear partial differential equations
This work is concerned with the use of Gaussian surrogate models for Bayesian inverse problems associated with linear partial differential equations. A particular focus is on the regime where only a small amount of train…
Gaussian ProcessesMeta-Learning Bayesian Neural Network Priors Based on PAC-Bayesian Theory
Bayesian deep learning is a promising approach towards improved uncertainty quantification and sample efficiency. Due to their complex parameter space, choosing informative priors for Bayesian Neural Networks (BNNs) is c…
Meta-LearningUncertainty QuantificationVariational InferenceGeneralized Variational Inference: Three arguments for deriving new Posteriors
We advocate an optimization-centric view on and introduce a novel generalization of Bayesian inference. Our inspiration is the representation of Bayes' rule as infinite-dimensional optimization problem (Csiszar, 1975; Do…
Bayesian InferenceGaussian ProcessesVariational Inference