Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference
Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations.
Code (1)
Tasks
Bayesian InferenceDiagnosticInductive BiasSimilar Papers 제목 키워드 기반
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem,…
Meta-LearningPredictionTranslationVIPriors 2: Visual Inductive Priors for Data-Efficient Deep Learning Challenges
The second edition of the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" challenges featured five data-impaired challenges, where models are trained from scratch on a reduced number of training samp…
Data AugmentationDeep LearningTransfer LearningC-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confounder
Representation learning assumes that real-world data is generated by a few semantically meaningful generative factors (i.e., sources of variation) and aims to discover them in the latent space. These factors are expected…
DisentanglementInductive BiasRepresentation LearningListening to the Noise: Blind Denoising with Gibbs Diffusion
In recent years, denoising problems have become intertwined with the development of deep generative models. In particular, diffusion models are trained like denoisers, and the distribution they model coincide with denois…
Bayesian InferenceDenoisingDiagnosticDeep Audio Priors Emerge From Harmonic Convolutional Networks
Convolutional neural networks (CNNs) excel in image recognition and generation. Among many efforts to explain their effectiveness, experiments show that CNNs carry strong inductive biases that capture natural image prior…