DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization
Adaptive gradient-based optimization methods such as \textsc{Adagrad}, \textsc{Rmsprop}, and \textsc{Adam} are widely used in solving large-scale machine learning problems including deep learning. A number of schemes have been proposed in the literature aiming at parallelizing them, based on communications of peripheral nodes with a central node, but incur high communications cost. To address this issue, we develop a novel consensus-based distributed adaptive moment estimation method (\textsc{Dadam}) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. The method is particularly useful, since it can accommodate settings where access to local data is allowed. Further, as established theoretically in this work, it can outperform centralized adaptive algorithms, for certain classes of loss functions used in applications. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that \textsc{Dadam} works also well in practice and compares favorably to competing online optimization methods.
Code (1)
Tasks
Stochastic OptimizationSimilar Papers 제목 키워드 기반
Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization
Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suf…
Federated LearningLDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during trai…
Stochastic Gradient Sampling for Enhancing Neural Networks Training
In this paper, we introduce StochGradAdam, a novel optimizer designed as an extension of the Adam algorithm, incorporating stochastic gradient sampling techniques to improve computational efficiency while maintaining rob…
Computational Efficiencyimage-classificationImage ClassificationAccelerating Fair Federated Learning: Adaptive Federated Adam
Federated learning is a distributed and privacy-preserving approach to train a statistical model collaboratively from decentralized data of different parties. However, when datasets of participants are not independent an…
FairnessFederated LearningPrivacy PreservingExtended Version of "Distributed Adaptive Resilient Consensus Control for Uncertain Nonlinear Multiagent Systems Against Deception Attacks"
This paper studies distributed resilient consensus problem for a class of uncertain nonlinear multiagent systems susceptible to deception attacks. The attacks invade both sensor and actuator channels of each agent. A spe…