Distributed Dual Averaging In Networks
The goal of decentralized optimization over a network is to optimize a global objective formed by a sum of local (possibly nonsmooth) convex functions using only local computation and communication. We develop and analyze distributed algorithms based on dual averaging of subgradients, and we provide sharp bounds on their convergence rates as a function of the network size and topology. Our analysis clearly separates the convergence of the optimization algorithm itself from the effects of communication constraints arising from the network structure. We show that the number of iterations required by our algorithm scales inversely in the spectral gap of the network. The sharpness of this prediction is confirmed both by theoretical lower bounds and simulations for various networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Adding vs. Averaging in Distributed Primal-Dual Optimization
Distributed optimization methods for large-scale machine learning suffer from a communication bottleneck. It is difficult to reduce this bottleneck while still efficiently and accurately aggregating partial work from dif…
Distributed OptimizationDistributed Convex Optimization With Limited Communications
In this paper, a distributed convex optimization algorithm, termed \emph{distributed coordinate dual averaging} (DCDA) algorithm, is proposed. The DCDA algorithm addresses the scenario of a large distributed optimization…
Distributed OptimizationvalidFederated Composite Optimization
Federated Learning (FL) is a distributed learning paradigm that scales on-device learning collaboratively and privately. Standard FL algorithms such as FedAvg are primarily geared towards smooth unconstrained settings. I…
Federated LearningAccelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum
Momentum method has been used extensively in optimizers for deep learning. Recent studies show that distributed training through K-step averaging has many nice properties. We propose a momentum method for such model aver…
Trading Redundancy for Communication: Speeding up Distributed SGD for Non-convex Optimization
Communication overhead is one of the key challenges that hinder the scalability of distributed optimization algorithms to train large neural networks. In recent years, there has been a great deal of research to alleviate…
Distributed OptimizationDiversity