SoftAdam: Unifying SGD and Adam for better stochastic gradient descent
Abstract Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
On Higher-order Moments in Adam
In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we …
Variational Stochastic Gradient Descent for Deep Neural Networks
Current state-of-the-art optimizers are adaptive gradient-based optimization methods such as Adam. Recently, there has been an increasing interest in formulating gradient-based optimizers in a probabilistic framework for…
image-classificationImage ClassificationVariational InferenceStochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, m…
General Classificationimage-classificationLanguage Modelingspeech-recognition+2Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, m…
General Classificationimage-classificationImage ClassificationLanguage Modeling+5Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoretical understanding of this discrepancy i…