Dual Averaging is Surprisingly Effective for Deep Learning Optimization

2020-10-20 · Samy Jelassi, Aaron Defazio

First-order stochastic optimization methods are currently the most widely used class of methods for training deep neural networks. However, the choice of the optimizer has become an ad-hoc rule that can significantly affect the performance. For instance, SGD with momentum (SGD+M) is typically used in computer vision (CV) and Adam is used for training transformer models for Natural Language Processing (NLP). Using the wrong method can lead to significant performance degradation. Inspired by the dual averaging algorithm, we propose Modernized Dual Averaging (MDA), an optimizer that is able to perform as well as SGD+M in CV and as Adam in NLP. Our method is not adaptive and is significantly simpler than Adam. We show that MDA induces a decaying uncentered $L_2$-regularization compared to vanilla SGD+M and hypothesize that this may explain why it works on NLP problems where SGD+M fails.

📄 PDF Abstract BibTeX arXiv:2010.10502

Code (0)

등록된 구현이 없습니다.

Tasks

Deep LearningStochastic Optimization

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

SGD with Momentum 설명 없음

Adam 설명 없음

Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization

2020-12-29 · Wei Tao, Wei Li, Zhisong Pan, Qing Tao

Averaging scheme has attracted extensive attention in deep learning as well as traditional machine learning. It achieves theoretically optimal convergence and also improves the empirical model performance. However, there…

Unifying mirror descent and dual averaging

2019-10-30 · Anatoli Juditsky, Joon Kwon, Éric Moulines

We introduce and analyze a new family of first-order optimization algorithms which generalizes and unifies both mirror descent and dual averaging. Within the framework of this family, we define new algorithms for constra…

Federated Composite Optimization

2020-11-17 · Honglin Yuan, Manzil Zaheer, Sashank Reddi

Federated Learning (FL) is a distributed learning paradigm that scales on-device learning collaboratively and privately. Standard FL algorithms such as FedAvg are primarily geared towards smooth unconstrained settings. I…

Federated Learning

Snapshot Ensembles: Train 1, get M for free

2017-04-01 · Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu 외

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a …

Composite Optimization with Error Feedback: the Dual Averaging Approach

2025-10-03 · Yuan Gao, Anton Rodomanov, Jeremy Rack, Sebastian Stich arxiv

Communication efficiency is a central challenge in distributed machine learning training, and message compression is a widely used solution. However, standard Error Feedback (EF) methods (Seide et al., 2014), though effe…