A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum with a large neighboring region in the parameter space from which each weight vector has similar small error. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness, gradient noise and hyperparameters. We verify an interesting fact that the stochastic gradient noise covariance is nearly proportional to the Hessian and inverse to the batch size near minima. We prove that, benefited from stochastic gradient noise, SGD favors flat minima exponentially more than sharp minima, while Gradient Descent with injected white noise favors flat minima only polynomially more than sharp minima. We also prove that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of batch size and learning rate, and thus cannot search flat minima efficiently in a realistic computational time.
Code (0)
등록된 구현이 없습니다.
Tasks
Stochastic OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the diffusion approximation of nonconvex stochastic gradient descent
We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD…
Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features
We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing …
Learning TheoryStochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stocha…
A note on diffusion limits for stochastic gradient descent
In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in st…
Stochastic Gradient Descent over P2
Stochastic gradient descent (SGD) admits diffusion approximations that replace the complicated randomness of stochastic gradients by Gaussian noise, providing a powerful tool for understanding its dynamics and long-time …
Stochastic Optimization