SGD Can Converge to Local Maxima

2021-09-29 · ICLR 2022 4 · Liu Ziyin, Botao Li, James B Simon, Masahito Ueda

Stochastic gradient descent (SGD) is widely used for the nonlinear, nonconvex problem of training deep neural networks, but its behavior remains poorly understood. Many theoretical works have studied SGD, but they commonly rely on restrictive and unrealistic assumptions about the nature of its noise. In this work, we construct example optimization problems illustrating that, if these assumptions are relaxed, SGD can exhibit many strange behaviors that run counter to the established wisdom of the field. Our constructions show that (1) SGD can converge to local maxima, (2) SGD might only escape saddle points arbitrarily slowly, (3) SGD can prefer sharp minima over flat ones, and (4) AMSGrad can converge to local maxima. We realize our most surprising results in a simple neural network-like construction, suggesting their relevance to deep learning.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

AMSGrad AMSGrad is a stochastic optimization method that seeks to fix a convergence issue with Adam based optimizers. AMSGrad uses the…

SGD with a Constant Large Learning Rate Can Converge to Local Maxima

2021-07-25 · Liu Ziyin, Botao Li, James B. Simon, Masahito Ueda

Previous works on stochastic gradient descent (SGD) often focus on its success. In this work, we construct worst-case optimization problems illustrating that, when not in the regimes that the previous works often assume,…

Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences

2016-09-04 · NeurIPS 2016 12 · Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin J. Wainwright 외

We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with $M \geq 3$ components. Our first main result shows that the population likelihood function has ba…

Open-Ended Question Answering

On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks

2018-07-11 · Hoang Thanh-Tung, Truyen Tran

In this paper, we show that Generative Adversarial Networks (GANs) suffer from catastrophic forgetting even when they are trained to approximate a single target distribution. We show that GAN training is a continual lear…

Continual Learning

Approximation of Maximally Monotone Operators : A Graph Convergence Perspective

2026-05-12 · Takashi Furuya, Yury Korolev, Takaharu Yaguchi arxiv

Operator learning has been highly successful for continuous mappings between infinite-dimensional spaces, such as PDE solution operators. However, many operators of interest-including differential operators-are discontin…

Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature

2021-02-08 · NeurIPS 2021 12 · Kefan Dong, Jiaqi Yang, Tengyu Ma

This paper studies model-based bandit and reinforcement learning (RL) with nonlinear function approximations. We propose to study convergence to approximate local maxima because we show that global convergence is statist…

reinforcement-learning Reinforcement Learning (RL)