Convergence of gradient descent for deep neural networks
This article presents a criterion for convergence of gradient descent to a global minimum, which is then used to show that gradient descent with proper initialization converges to a global minimum when training any feedforward neural network with smooth and strictly increasing activation functions, provided that the input dimension is greater than or equal to the number of data points. The main difference with prior work is that the width of the network can be a fixed number instead of growing as some multiple or power of the number of data points.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent
Factorization-based gradient descent is a scalable and efficient algorithm for solving low-rank matrix completion. Recent progress in structured non-convex optimization has offered global convergence guarantees for gradi…
Low-Rank Matrix CompletionMatrix CompletionLinear Convergence of Generalized Mirror Descent with Time-Dependent Mirrors
The Polyak-Lojasiewicz (PL) inequality is a sufficient condition for establishing linear convergence of gradient descent, even in non-convex settings. While several recent works use a PL-based analysis to establish linea…
Global Convergence Analysis of Vanilla Gradient Descent for Asymmetric Matrix Completion
This paper investigates the asymmetric low-rank matrix completion problem, which can be formulated as an unconstrained non-convex optimization problem with a nonlinear least-squares objective function, and is solved via …
Convergence Analysis of Fractional Gradient Descent
Fractional derivatives are a well-studied generalization of integer order derivatives. Naturally, for optimization, it is of interest to understand the convergence properties of gradient descent using fractional derivati…
A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality
Adaptive gradient-descent optimizers are the standard choice for training neural network models. Despite their faster convergence than gradient-descent and remarkable performance in practice, the adaptive optimizers are …