Adaptively Preconditioned Stochastic Gradient Langevin Dynamics

2019-06-10 · Chandrasekaran Anirudh Bhardwaj

Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed to precondition the noise in order to achieve better convergence. In this paper, we describe an adaptive method to estimate the parameters of the noise and conduct experiments on well-known model architectures to show that the adaptively preconditioned SGLD method achieves convergence with the speed of adaptive first order methods such as Adam, AdaGrad etc. and achieves generalization equivalent of SGD in the test set.

📄 PDF Abstract BibTeX arXiv:1906.04324

Code (1)

Anirudhsekar96/Noisy_SGD pytorch

Tasks

Navigate

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Adam 설명 없음

AdaGrad AdaGrad is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts

2020-09-20 · Sehwan Kim, Qifan Song, Faming Liang

Bayesian deep learning offers a principled way to address many issues concerning safety of artificial intelligence (AI), such as model uncertainty,model interpretability, and prediction bias. However, due to the lack of …

Deep Learning

Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

2015-12-23 · Chunyuan Li, Changyou Chen, David Carlson, Lawrence Carin

Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace and more

Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications

2020-06-29 · Yating Wang, Wei Deng, Lin Guang

In this work, we propose a Bayesian type sparse deep learning algorithm. The algorithm utilizes a set of spike-and-slab priors for the parameters in the deep neural network. The hierarchical Bayesian mixture will be trai…

Sparse Learning

Natural Langevin Dynamics for Neural Networks

2017-12-04 · Gaétan Marceau-Caron, Yann Ollivier

One way to avoid overfitting in machine learning is to use model parameters distributed according to a Bayesian posterior given the data, rather than the maximum likelihood estimator. Stochastic gradient Langevin dynamic…

Adaptive Non-reversible Stochastic Gradient Langevin Dynamics

2020-09-26 · Vikram Krishnamurthy, George Yin

It is well known that adding any skew symmetric matrix to the gradient of Langevin dynamics algorithm results in a non-reversible diffusion with improved convergence rate. This paper presents a gradient algorithm to adap…