Representing Model Uncertainty of Neural Networks in Sparse Information Form
This paper addresses the problem of representing a system's belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN). The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs. To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form. Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank approximation of this eigenbasis and a memory-efficient sampling scheme. We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
FormSimilar Papers 제목 키워드 기반
Uncertainty Weighted Causal Graphs
Causality has traditionally been a scientific way to generate knowledge by relating causes to effects. From an imaginery point of view, causal graphs are a helpful tool for representing and infering new causal informatio…
ManagementImproving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
Standard sparse pseudo-input approximations to the Gaussian process (GP) cannot handle complex functions well. Sparse spectrum alternatives attempt to answer this but are known to over-fit. We suggest the use of variatio…
Variational InferenceEUBRL: Epistemic Uncertainty Directed Bayesian Reinforcement Learning
At the boundary between the known and the unknown, an agent inevitably confronts the dilemma of whether to explore or to exploit. Epistemic uncertainty reflects such boundaries, representing systematic uncertainty due to…
Reinforcement LearningDropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematical…
Bayesian InferenceDeep Reinforcement LearningGaussian ProcessesGeneral Classification+3Estimating Model Uncertainty of Neural Network in Sparse Information Form
We present a sparse representation of model uncertainty for Deep Neural Networks (DNNs) where the parameter posterior is approximated with an inverse formulation of the Multivariate Normal Distribution (MND), also known …
Dimensionality ReductionForm