Local SGD With a Communication Overhead Depending Only on the Number of Workers

2020-06-03 · Artin Spiridonoff, Alex Olshevsky, Ioannis Ch. Paschalidis

We consider speeding up stochastic gradient descent (SGD) by parallelizing it across multiple workers. We assume the same data set is shared among $n$ workers, who can take SGD steps and coordinate with a central server. Unfortunately, this could require a lot of communication between the workers and the server, which can dramatically reduce the gains from parallelism. The Local SGD method, proposed and analyzed in the earlier literature, suggests machines should make many local steps between such communications. While the initial analysis of Local SGD showed it needs $\Omega ( \sqrt{T} )$ communications for $T$ local gradient steps in order for the error to scale proportionately to $1/(nT)$, this has been successively improved in a string of papers, with the state-of-the-art requiring $\Omega \left( n \left( \mbox{ polynomial in log } (T) \right) \right)$ communications. In this paper, we give a new analysis of Local SGD. A consequence of our analysis is that Local SGD can achieve an error that scales as $1/(nT)$ with only a fixed number of communications independent of $T$: specifically, only $\Omega(n)$ communications are required.

📄 PDF Abstract BibTeX arXiv:2006.02582

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Local SGD Local SGD is a distributed training technique that runs SGD independently in parallel on different workers and averages the sequences…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Distributed Distillation for On-Device Learning

2020-12-01 · NeurIPS 2020 12 · Ilai Bistritz, Ariana Mann, Nicholas Bambos

On-device learning promises collaborative training of machine learning models across edge devices without the sharing of user data. In state-of-the-art on-device learning algorithms, devices communicate their model weigh…

FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning

2021-09-02 · Zhifeng Jiang, Wei Wang, Yang Liu

Homomorphic encryption (HE) is a promising privacy-preserving technique for cross-silo federated learning (FL), where organizations perform collaborative model training on decentralized data. Despite the strong privacy g…

Federated Learning Privacy Preserving

Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

2026-05-31 · Tehila Dahan, Bassel Hamoud, Roie Reshef, Martin Jaggi 외 arxiv

Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still…

Distributed Optimization

Horovod: fast and easy distributed deep learning in TensorFlow

2018-02-15 · Alexander Sergeev, Mike Del Balso

Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complicat…

Deep Learning GPU

Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Range

2024-03-05 · Yuzi Yan, Yuan Shen

This paper proposes a scalable distributed policy gradient method and proves its convergence to near-optimal solution in multi-agent linear quadratic networked systems. The agents engage within a specified network under …