On the Inconsistency of Bayesian Inference for Misspecified Neural Networks
Grunwald and Van Ommen (2017) show that Bayesian inference for linear regression can be inconsistent under model misspecification. In this paper, we extend their analysis to Bayesian neural networks (BNNs), investigating if they too can be inconsistent under misspecification. We find that BNNs exhibit the same inconsistency when Hamiltonian Monte Carlo is used for posterior inference. However, variational inference changes this behavior. Surprisingly, we find that variational Bayes leads to BNNs that are consistent in the setting studied by Grunwald and Van Ommen (2017). We conjecture that the success of variational Bayes is due to its optimization objective: the evidence lower bound (ELBO) implicitly encourages the posterior approximation to concentrate, mitigating the ill-effects of the misspecification.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceregressionVariational InferenceSimilar Papers 제목 키워드 기반
Practical calibration of the temperature parameter in Gibbs posteriors
PAC-Bayesian algorithms and Gibbs posteriors are gaining popularity due to their robustness against model misspecification even when Bayesian inference is inconsistent. The PAC-Bayesian alpha-posterior is a generalizatio…
Bayesian InferenceSemi-Modular Inference: enhanced learning in multi-modular models by tempering the influence of components
Bayesian statistical inference loses predictive optimality when generative models are misspecified. Working within an existing coherent loss-based generalisation of Bayesian inference, we show existing Modular/Cut-model …
Bayesian InferenceMeta-LearningGeneralized Bayesian Inference for Scientific Simulators via Amortized Cost Estimation
Simulation-based inference (SBI) enables amortized Bayesian inference for simulators with implicit likelihoods. But when we are primarily interested in the quality of predictive simulations, or when the model cannot exac…
Bayesian InferenceNon-Bayesian Learning in Misspecified Models
Deviations from Bayesian updating are traditionally categorized as biases, errors, or fallacies, thus implying their inherent ``sub-optimality.'' We offer a more nuanced view. We demonstrate that, in learning problems wi…
Generalized Laplace Approximation
In recent years, the inconsistency in Bayesian deep learning has garnered increasing attention. Tempered or generalized posterior distributions often offer a direct and effective solution to this issue. However, understa…
Attribute