paper-with-me

홈 › Papers

A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima

2020-02-10 · ICLR 2021 1 · Zeke Xie, Issei Sato, Masashi Sugiyama

Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum with a large neighboring region in the parameter space from which each weight vector has similar small error. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness, gradient noise and hyperparameters. We verify an interesting fact that the stochastic gradient noise covariance is nearly proportional to the Hessian and inverse to the batch size near minima. We prove that, benefited from stochastic gradient noise, SGD favors flat minima exponentially more than sharp minima, while Gradient Descent with injected white noise favors flat minima only polynomially more than sharp minima. We also prove that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of batch size and learning rate, and thus cannot search flat minima efficiently in a realistic computational time.

📄 PDF Abstract BibTeX arXiv:2002.03495

Code (0)

등록된 구현이 없습니다.

Tasks

Stochastic Optimization

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Similar Papers 제목 키워드 기반

On the diffusion approximation of nonconvex stochastic gradient descent

2017-05-22 · Wenqing Hu, Chris Junchi Li, Lei LI, Jian-Guo Liu

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD…

Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features

2024-02-12 · Rodrigo Veiga, Anastasia Remizova, Nicolas Macris

We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing …

Learning Theory

Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

2023-02-14 · Benjamin Gess, Sebastian Kassing, Vitalii Konarovskyi

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stocha…

A note on diffusion limits for stochastic gradient descent

2022-10-20 · Alberto Lanconelli, Christopher S. A. Lauria

In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in st…

Stochastic Gradient Descent over P2

2026-09-11 · Maria Oprea, Qin Li, Yunan Yang arxiv

Stochastic gradient descent (SGD) admits diffusion approximations that replace the complicated randomness of stochastic gradients by Gaussian noise, providing a powerful tool for understanding its dynamics and long-time …

Stochastic Optimization