Gradient Descent Algorithm Survey
Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey
This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start wit…
Distributed OptimizationMultiobjective OptimizationSecond-order methodsStochastic OptimizationSurvey on Algorithms for multi-index models
We review the literature on algorithms for estimating the index space in a multi-index model. The primary focus is on computationally efficient (polynomial-time) algorithms in Gaussian space, the assumptions under which …
SurveySurvey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization
For strongly convex objectives that are smooth, the classical theory of gradient descent ensures linear convergence relative to the number of gradient evaluations. An analogous nonsmooth theory is challenging. Even when …
Survey schemes for stochastic gradient descent with applications to M-estimation
In certain situations that shall be undoubtedly more and more common in the Big Data era, the datasets available are so massive that computing statistics over the full sample is hardly feasible, if not unfeasible. A natu…
SurveySurvey SamplingA Survey of Algorithms and Analysis for Adaptive Online Learning
We present tools for the analysis of Follow-The-Regularized-Leader (FTRL), Dual Averaging, and Mirror Descent algorithms when the regularizer (equivalently, prox-function or learning rate schedule) is chosen adaptively b…
Survey