A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
Gradient descent optimization algorithms are the standard ingredients that are used to train artificial neural networks (ANNs). Even though a huge number of numerical simulations indicate that gradient descent optimization methods do indeed convergence in the training of ANNs, until today there is no rigorous theoretical analysis which proves (or disproves) this conjecture. In particular, even in the case of the most basic variant of gradient descent optimization algorithms, the plain vanilla gradient descent method, it remains an open problem to prove or disprove the conjecture that gradient descent converges in the training of ANNs. In this article we solve this problem in the special situation where the target function under consideration is a constant function. More specifically, in the case of constant target functions we prove in the training of rectified fully-connected feedforward ANNs with one-hidden layer that the risk function of the gradient descent method does indeed converge to zero. Our mathematical analysis strongly exploits the property that the rectifier function is the activation function used in the considered ANNs. A key contribution of this work is to explicitly specify a Lyapunov function for the gradient flow system of the ANN parameters. This Lyapunov function is the central tool in our convergence proof of the gradient descent method.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
In this article we study the stochastic gradient descent (SGD) optimization method in the training of fully-connected feedforward artificial neural networks with ReLU activation. The main result of this work proves that …
The convergence of the Stochastic Gradient Descent (SGD) : a self-contained proof
We give here a proof of the convergence of the Stochastic Gradient Descent (SGD) in a self-contained manner.
On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths
We give a simple proof for the global convergence of gradient descent in training deep ReLU networks with the standard square loss, and show some of its improvements over the state-of-the-art. In particular, while prior …
On the modes of convergence of Stochastic Optimistic Mirror Descent (OMD) for saddle point problems
In this article, we study the convergence of Mirror Descent (MD) and Optimistic Mirror Descent (OMD) for saddle point problems satisfying the notion of coherence as proposed in Mertikopoulos et al. We prove convergence o…
Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases
In recent years, artificial neural networks have developed into a powerful tool for dealing with a multitude of problems for which classical solution approaches reach their limits. However, it is still unclear why random…