paper-with-me

홈 › Papers

On the Origin of Implicit Regularization in Stochastic Gradient Descent

2021-01-28 · ICLR 2021 1 · Samuel L. Smith, Benoit Dherin, David G. T. Barrett, Soham De

For infinitesimal learning rates, stochastic gradient descent (SGD) follows the path of gradient flow on the full batch loss function. However moderately large learning rates can achieve higher test accuracies, and this generalization benefit is not explained by convergence bounds, since the learning rate which maximizes test accuracy is often larger than the learning rate which minimizes training loss. To interpret this phenomenon we prove that for SGD with random shuffling, the mean SGD iterate also stays close to the path of gradient flow if the learning rate is small and finite, but on a modified loss. This modified loss is composed of the original loss function and an implicit regularizer, which penalizes the norms of the minibatch gradients. Under mild assumptions, when the batch size is small the scale of the implicit regularization term is proportional to the ratio of the learning rate to the batch size. We verify empirically that explicitly including the implicit regularizer in the loss can enhance the test accuracy when the learning rate is small.

📄 PDF Abstract BibTeX arXiv:2101.12176

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Similar Papers 제목 키워드 기반

Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent

2023-02-02 · Avrajit Ghosh, He Lyu, Xitong Zhang, Rongrong Wang

It is well known that the finite step-size ($h$) in Gradient Descent (GD) implicitly regularizes solutions to flatter minima. A natural question to ask is "Does the momentum parameter $\beta$ play a role in implicit regu…

Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization

2018-06-04 · ICLR 2019 5 · Navid Azizan, Babak Hassibi

Stochastic descent methods (of the gradient and mirror varieties) have become increasingly popular in optimization. In fact, it is now widely recognized that the success of deep learning is not only due to the special de…

Implicit biases in multitask and continual learning from a backward error analysis perspective

2023-11-01 · Benoit Dherin

Using backward error analysis, we compute implicit training biases in multitask and continual learning settings for neural networks trained with stochastic gradient descent. In particular, we derive modified losses that …

Continual Learning

A note on diffusion limits for stochastic gradient descent

2022-10-20 · Alberto Lanconelli, Christopher S. A. Lauria

In the machine learning literature stochastic gradient descent has recently been widely discussed for its purported implicit regularization properties. Much of the theory, that attempts to clarify the role of noise in st…

The Implicit Regularization of Stochastic Gradient Flow for Least Squares

2020-03-17 · ICML 2020 1 · Alnur Ali, Edgar Dobriban, Ryan J. Tibshirani

We study the implicit regularization of mini-batch stochastic gradient descent, when applied to the fundamental problem of least squares regression. We leverage a continuous-time stochastic differential equation having t…

regression