A visual exploration of Gaussian Processes and Infinite Neural Networks
Gaussian Processes generalize Gaussian distributions to functons as random variables (stochastic processes). As such, they are a powerful tool for regression and Bayesian inference. In the last years, they have received increased attention due to the interest in machine learning. An interesting connection between Gaussian Processes and Neural Networks exists when the width of feed-forward neural networks tends towards infinity. For a specific choice of priors over the network parameters, the resulting network samples will be distributed according to a Gaussian Process. In this blog post, I intend to explore this connection between Gaussian Processes and infinite Neural Nets, using Julia code and visual examples. I believe that there is some value in creating an intuitive understanding of the connection between those two methods as Gaussian Processes may help us understand how well Neural Networks can perform in the limit of infinite neurons per layer. Moreover, they may do so while keeping the probabilistic nature of the process in mind, as network training and the resulting performance will heavily depend on the initialization of the network weights. Finally, I discuss the limitations of this approach and how it may be extended by using Bayesian Neural Networks.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceGaussian ProcessesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Optimal Reinforcement Learning for Gaussian Systems
The exploration-exploitation trade-off is among the central challenges of reinforcement learning. The optimal Bayesian solution is intractable in general. This paper studies to what extent analytic statements about optim…
Gaussian Processesreinforcement-learningReinforcement LearningReinforcement Learning (RL)Infinite Mixtures of Multivariate Gaussian Processes
This paper presents a new model called infinite mixtures of multivariate Gaussian processes, which can be used to learn vector-valued functions and applied to multitask learning. As an extension of the single multivariat…
Gaussian ProcessesregressionStudent-t processes as infinite-width limits of posterior Bayesian neural networks
The asymptotic properties of Bayesian Neural Networks (BNNs) have been extensively studied, particularly regarding their approximations by Gaussian processes in the infinite-width limit. We extend these results by showin…
Gaussian ProcessesDeep learning with differential Gaussian process flows
We propose a novel deep learning paradigm of differential flows that learn a stochastic differential equation transformations of inputs prior to a standard classification or regression function. The key property of diffe…
Deep LearningGaussian ProcessesGeneral ClassificationregressionScale Mixtures of Neural Network Gaussian Processes
Recent works have revealed that infinitely-wide feed-forward or recurrent neural networks of any architecture correspond to Gaussian processes referred to as Neural Network Gaussian Processes (NNGPs). While these works h…
Gaussian Processes