Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d} \log d)$ per-iteration communication cost at best, where $d$ is the size of the model. We propose hyper-sphere quantization (HSQ), a general framework that can be configured to achieve a continuum of trade-offs between communication efficiency and gradient accuracy. In particular, at the high compression ratio end, HSQ provides a low per-iteration communication cost of $O(\log d)$, which is favorable for federated learning. We prove the convergence of HSQ theoretically and show by experiments that HSQ significantly reduces the communication cost of model training without hurting convergence accuracy.
Code (1)
Tasks
Federated LearningQuantizationSimilar Papers 제목 키워드 기반
SphereFed: Hyperspherical Federated Learning
Federated Learning aims at training a global model from multiple decentralized devices (i.e. clients) without exchanging their private local data. A key challenge is the handling of non-i.i.d. (independent identically di…
Federated LearningAQUILA: Communication Efficient Federated Learning with Adaptive Quantization in Device Selection Strategy
The widespread adoption of Federated Learning (FL), a privacy-preserving distributed learning methodology, has been impeded by the challenge of high communication overheads, typically arising from the transmission of lar…
Federated LearningPrivacy PreservingQuantizationCommunication-Efficient Federated Learning via Clipped Uniform Quantization
This paper presents a novel approach to enhance communication efficiency in federated learning through clipped uniform quantization. By leveraging optimal clipping thresholds and client-specific adaptive quantization sch…
Federated LearningQuantizationDEED: A General Quantization Scheme for Communication Efficiency in Bits
In distributed optimization, a popular technique to reduce communication is quantization. In this paper, we provide a general analysis framework for inexact gradient descent that is applicable to quantization schemes. We…
Distributed OptimizationFederated LearningQuantizationFedDQ: Communication-Efficient Federated Learning with Descending Quantization
Federated learning (FL) is an emerging learning paradigm without violating users' privacy. However, large model size and frequent model aggregation cause serious communication bottleneck for FL. To reduce the communicati…
Federated LearningModel CompressionPrivacy PreservingQuantization