Stabilising priors for robust Bayesian deep learning
Bayesian neural networks (BNNs) have developed into useful tools for probabilistic modelling due to recent advances in variational inference enabling large scale BNNs. However, BNNs remain brittle and hard to train, especially: (1) when using deep architectures consisting of many hidden layers and (2) in situations with large weight variances. We use signal propagation theory to quantify these challenges and propose self-stabilising priors. This is achieved by a reformulation of the ELBO to allow the prior to influence network signal propagation. Then, we develop a stabilising prior, where the distributional parameters of the prior are adjusted before each forward pass to ensure stability of the propagating signal. This stabilised signal propagation leads to improved convergence and robustness making it possible to train deeper networks and in more noisy settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningVariational InferenceSimilar Papers 제목 키워드 기반
Tipping Cycles
Ecological systems are studied using many different approaches and mathematical tools. One approach, based on the Jacobian of Lotka-Volterra type models, has been a staple of mathematical ecology for years, leading to ma…
Priors in Bayesian Deep Learning: A Review
While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we …
Bayesian InferenceDeep LearningGaussian ProcessesExact marginal prior distributions of finite Bayesian neural networks
Bayesian neural networks are theoretically well-understood only in the infinite-width limit, where Gaussian priors over network weights yield Gaussian priors over network outputs. Recent work has suggested that finite Ba…
BNNpriors: A library for Bayesian neural network inference with different prior distributions
Bayesian neural networks have shown great promise in many applications where calibrated uncertainty estimates are crucial and can often also lead to a higher predictive performance. However, it remains challenging to cho…
Osmolyte-Induced Protein Stability Changes Explained by Graph Theory
Enhanced stabilisation of protein structures via the presence of inert excipients is a key mechanism adopted both by physiological systems and in biotechnological applications. While the intrinsic stability of proteins i…