Model Selection in Bayesian Neural Networks via Horseshoe Priors
Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.
Code (1)
Tasks
Model SelectionOpen-Ended Question AnsweringSimilar Papers 제목 키워드 기반
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an …
Model SelectionOpen-Ended Question Answeringreinforcement-learningReinforcement Learning+1Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods
Choosing between classical and Bayesian sparse regression methods involves a real trade-off: penalized estimators like Lasso run in milliseconds but give no uncertainty estimates,while Horseshoe and Spike-and-Slab priors…
Semi-parametric Expert Bayesian Network Learning with Gaussian Processes and Horseshoe Priors
This paper proposes a model learning Semi-parametric rela- tionships in an Expert Bayesian Network (SEBN) with linear parameter and structure constraints. We use Gaussian Pro- cesses and a Horseshoe prior to introduce mi…
Gaussian ProcessesFalse Discovery Rate Control via Frequentist-assisted Horseshoe
The horseshoe prior, a widely used handy alternative to the spike-and-slab prior, has proven to be an exceptional default global-local shrinkage prior in Bayesian inference and machine learning. However, designing tests …
Bayesian InferenceHorseshoe Regularization for Machine Learning in Complex and Deep Models
Since the advent of the horseshoe priors for regularization, global-local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high-dimens…
BIG-bench Machine Learningregression