Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks

2022-01-28 · Bartłomiej Polaczyk, Jacek Cyranka

We study the overparametrization bounds required for the global convergence of stochastic gradient descent algorithm for a class of one hidden layer feed-forward neural networks, considering most of the activation functions used in practice, including ReLU. We improve the existing state-of-the-art results in terms of the required hidden layer width. We introduce a new proof technique combining nonlinear analysis with properties of random initializations of the network. First, we establish the global convergence of continuous solutions of the differential inclusion being a nonsmooth analogue of the gradient flow for the MSE loss. Second, we provide a technical result (working also for general approximators) relating solutions of the aforementioned differential inclusion to the (discrete) stochastic gradient descent sequences, hence establishing linear convergence towards zero loss for the stochastic gradient descent iterations.

📄 PDF Abstract BibTeX arXiv:2201.12052

Code (0)

등록된 구현이 없습니다.

Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems

2023-09-21 · Nathan Buskulic, Jalal Fadili, Yvain Quéau

Neural networks have become a prominent approach to solve inverse problems in recent years. While a plethora of such methods was developed to solve inverse problems empirically, we are still lacking clear theoretical gua…

Convergence Guarantees of Overparametrized Wide Deep Inverse Prior

2023-03-20 · Nathan Buskulic, Yvain Quéau, Jalal Fadili

Neural networks have become a prominent approach to solve inverse problems in recent years. Amongst the different existing methods, the Deep Image/Inverse Priors (DIPs) technique is an unsupervised approach that optimize…

Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration

2018-09-12 · Xuefeng Gao, Mert Gürbüzbalaban, Lingjiong Zhu

Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is a variant of stochastic gradient with momentum where a controlled and properly scaled Gaussian noise is added to the stochastic gradients to steer the iterates towar…

Stochastic Optimization

Finite Sample Convergence Rates of Zero-Order Stochastic Optimization Methods

2012-12-01 · NeurIPS 2012 12 · Andre Wibisono, Martin J. Wainwright, Michael. I. Jordan, John C. Duchi

We consider derivative-free algorithms for stochastic optimization problems that use only noisy function values rather than gradients, analyzing their finite-sample convergence rates. We show that if pairs of function va…

Stochastic Optimization

Neural Temporal-Difference Learning Converges to Global Optima

2019-12-01 · NeurIPS 2019 12 · Qi Cai, Zhuoran Yang, Jason D. Lee, Zhaoran Wang

Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coup…

Deep Reinforcement Learning Q-Learning Reinforcement Learning Reinforcement Learning (RL)