LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression

2024-03-07 · Laurent Condat, Artavazd Maranjyan, Peter Richtárik

In Distributed optimization and Learning, and even more in the modern framework of federated learning, communication, which is slow and costly, is critical. We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats. LoCoDL works with a large class of unbiased compressors that includes widely-used sparsification and quantization methods. LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions. This is confirmed in practice, with LoCoDL outperforming existing algorithms.

📄 PDF Abstract BibTeX arXiv:2403.04348

Code (0)

등록된 구현이 없습니다.

Tasks

Distributed Optimization Federated Learning Quantization

Deep Gradient Compression Reduce the Communication Bandwidth For distributed Traning

2017-12-05 · The International Conference on Learning Representations 2017 12 · Yujun Lin, Song Han, Huizi Mao, Yu Wang 외

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The si…

Federated Learning image-classification Image Classification Language Modeling+3

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

2017-12-05 · ICLR 2018 1 · Yujun Lin, Song Han, Huizi Mao, Yu Wang 외

Federated Learning image-classification Image Classification Language Modeling+3

Communication-Efficient Distributed Learning with Local Immediate Error Compensation

2024-02-19 · Yifei Cheng, Li Shen, Linli Xu, Xun Qian 외

Gradient compression with error compensation has attracted significant attention with the target of reducing the heavy communication overhead in distributed learning. However, existing compression methods either perform …

TAMUNA: Doubly Accelerated Distributed Optimization with Local Training, Compression, and Partial Participation

2023-02-20 · Laurent Condat, Ivan Agarský, Grigory Malinovsky, Peter Richtárik

In distributed optimization and learning, several machines alternate between local computations in parallel and communication with a distant server. Communication is usually slow and costly and forms the main bottleneck.…

Distributed Optimization Federated Learning

Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

2025-09-10 · Kai Yi arxiv

Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores…

Federated Learning Model Compression