Sequential Function-Space Variational Inference via Gaussian Mixture Approximation
Continual learning is learning from a sequence of tasks with the aim of learning new tasks without forgetting old tasks. Sequential function-space variational inference (SFSVI) is a continual learning method based on variational inference which uses a Gaussian variational distribution to approximate the distribution of the outputs of a finite number of selected inducing points. Since the posterior distribution of a neural network is multi-modal, a Gaussian distribution could only match one mode of the posterior distribution, and a Gaussian mixture distribution could be used to better approximate the posterior distribution. We propose an SFSVI method which uses a Gaussian mixture variational distribution. We also compare different types of variational inference methods with and without a fixed pre-trained feature extractor. We find that in terms of final average accuracy, Gaussian mixture methods perform better than Gaussian methods and likelihood-focused methods perform better than prior-focused methods.
Code (1)
Tasks
Continual LearningVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning
We develop a framework for generalized variational inference in infinite-dimensional function spaces and use it to construct a method termed Gaussian Wasserstein inference (GWI). GWI leverages the Wasserstein distance be…
Gaussian ProcessesUncertainty QuantificationVariational InferenceVariational Gaussian Process State-Space Models
State-space models have been successfully used for more than fifty years in different areas of science and engineering. We present a procedure for efficient variational Bayesian learning of nonlinear state-space models b…
Gaussian ProcessesState Space ModelsTime SeriesTime Series Analysis+1Variational Inference for Gaussian Process Models with Linear Complexity
Large-scale Gaussian process inference has long faced practical challenges due to time and space complexity that is superlinear in dataset size. While sparse variational Gaussian process models are capable of learning fr…
Variational InferenceContinual Learning via Sequential Function-Space Variational Inference
Sequential Bayesian inference over predictive functions is a natural framework for continual learning from streams of data. However, applying it to neural networks has proved challenging in practice. Addressing the drawb…
Bayesian InferenceContinual LearningSequential Bayesian InferenceVariational InferenceContinual Multi-task Gaussian Processes
We address the problem of continual learning in multi-task Gaussian process (GP) models for handling sequential input-output observations. Our approach extends the existing prior-posterior recursion of online Bayesian in…
Bayesian InferenceContinual LearningGaussian ProcessesStochastic Optimization+1