Meta-Uncertainty in Bayesian Model Comparison
Bayesian model comparison (BMC) offers a principled probabilistic approach to study and rank competing models. In standard BMC, we construct a discrete probability distribution over the set of possible models, conditional on the observed data of interest. These posterior model probabilities (PMPs) are measures of uncertainty, but -- when derived from a finite number of observations -- are also uncertain themselves. In this paper, we conceptualize distinct levels of uncertainty which arise in BMC. We explore a fully probabilistic framework for quantifying meta-uncertainty, resulting in an applied method to enhance any BMC workflow. Drawing on both Bayesian and frequentist techniques, we represent the uncertainty over the uncertain PMPs via meta-models which combine simulated and observed data into a predictive distribution for PMPs on new data. We demonstrate the utility of the proposed method in the context of conjugate Bayesian regression, likelihood-based inference with Markov chain Monte Carlo, and simulation-based inference with neural networks.
Code (1)
Tasks
modelSimilar Papers 제목 키워드 기반
Bayesian meta learning for trustworthy uncertainty quantification
We consider the problem of Bayesian regression with trustworthy uncertainty quantification. We define that the uncertainty quantification is trustworthy if the ground truth can be captured by intervals dependent on the p…
Meta-LearningregressionUncertainty QuantificationALPaCA vs. GP-based Prior Learning: A Comparison between two Bayesian Meta-Learning Algorithms
Meta-learning or few-shot learning, has been successfully applied in a wide range of domains from computer vision to reinforcement learning. Among the many frameworks proposed for meta-learning, bayesian methods are part…
Few-Shot LearningMeta-LearningReinforcement Learning (RL)Bayesian Meta Sampling for Fast Uncertainty Adaptation
Meta learning has been making impressive progress for fast model adaptation. However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling. In this paper, we propose to achieve the goal …
Meta-LearningInformation-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning
The overall predictive uncertainty of a trained predictor can be decomposed into separate contributions due to epistemic and aleatoric uncertainty. Under a Bayesian formulation, assuming a well-specified model, the two c…
Meta-LearningBayesian Meta-Learning Through Variational Gaussian Processes
Recent advances in the field of meta-learning have tackled domains consisting of large numbers of small ("few-shot") supervised learning tasks. Meta-learning algorithms must be able to rapidly adapt to any individual few…
Gaussian ProcessesMeta-Learning