Communication-Efficient Local Decentralized SGD Methods
Recently, the technique of local updates is a powerful tool in centralized settings to improve communication efficiency via periodical communication. For decentralized settings, it is still unclear how to efficiently combine local updates and decentralized communication. In this work, we propose an algorithm named as LD-SGD, which incorporates arbitrary update schemes that alternate between multiple Local updates and multiple Decentralized SGDs, and provide an analytical framework for LD-SGD. Under the framework, we present a sufficient condition to guarantee the convergence. We show that LD-SGD converges to a critical point for a wide range of update schemes when the objective is non-convex and the training data are non-identically independent distributed. Moreover, our framework brings many insights into the design of update schemes for decentralized optimization. As examples, we specify two update schemes and show how they help improve communication efficiency. Specifically, the first scheme alternates the number of local and global update steps. From our analysis, the ratio of the number of local updates to that of decentralized SGD trades off communication and computation. The second scheme is to periodically shrink the length of local updates. We show that the decaying strategy helps improve communication efficiency both theoretically and empirically.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributed ComputingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Accelerating Decentralized Optimization via Overlapping Local Steps
Decentralized optimization has emerged as a critical paradigm for distributed learning, enabling scalable training while preserving data privacy through peer-to-peer collaboration. However, existing methods often suffer …
Decentralized Federated Learning: Balancing Communication and Computing Costs
Decentralized stochastic gradient descent (SGD) is a driving engine for decentralized federated learning (DFL). The performance of decentralized SGD is jointly influenced by inter-node communications and local updates. I…
Federated LearningMitigating Privacy-Utility Trade-off in Decentralized Federated Learning via $f$-Differential Privacy
Differentially private (DP) decentralized Federated Learning (FL) allows local users to collaborate without sharing their data with a central server. However, accurately quantifying the privacy budget of private FL algor…
Federated LearningLoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practic…
Image ClassificationFederated LearningA Proximal Gradient Method With Probabilistic Multi-Gossip Communications for Decentralized Composite Optimization
Decentralized optimization methods with local updates have recently gained attention for their provable ability to communication acceleration. In these methods, nodes perform several iterations of local computations betw…
Distributed Optimization