Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling
In this work we define a unified mathematical framework to deepen our understanding of the role of stochastic gradient (SG) noise on the behavior of Markov chain Monte Carlo sampling (SGMCMC) algorithms. Our formulation unlocks the design of a novel, practical approach to posterior sampling, which makes the SG noise isotropic using a fixed learning rate that we determine analytically, and that requires weaker assumptions than existing algorithms. In contrast, the common traits of existing \sgmcmc algorithms is to approximate the isotropy condition either by drowning the gradients in additive noise (annealing the learning rate) or by making restrictive assumptions on the \sg noise covariance and the geometry of the loss landscape. Extensive experimental validations indicate that our proposal is competitive with the state-of-the-art on \sgmcmc, while being much more practical to use.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling
Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: a…
Bayesian InferencePosterior Contraction Rates for Sparse Kolmogorov-Arnold Networks in Anisotropic Besov Spaces
We study posterior contraction rates for sparse Bayesian Kolmogorov-Arnold networks (KANs) over anisotropic Besov spaces, providing a statistical foundation of KANs from a Bayesian point of view. We show that sparse Baye…
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in com…
Decision MakingDeep Reinforcement LearningMulti-Armed Banditsreinforcement-learning+4Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Bounds
Understanding the generalization behavior of deep neural networks remains a fundamental challenge in modern statistical learning theory. Among existing approaches, PAC-Bayesian norm-based bounds have demonstrated particu…
Stochastic OptimizationWalking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks
Bayesian Neural Networks provide a principled framework for uncertainty quantification by modeling the posterior distribution of network parameters. However, exact posterior inference is computationally intractable, and …
Bayesian Inference