Double Quantization for Communication-Efficient Distributed Optimization
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. Three communication-efficient algorithms are proposed under this general scheme. Specifically, (i) we propose a low-precision algorithm AsyLPG with asynchronous parallelism, (ii) we explore integrating gradient sparsification with double quantization and develop Sparse-AsyLPG, (iii) we show that double quantization can also be accelerated by momentum technique and design accelerated AsyLPG. We establish rigorous performance guarantees for the algorithms, and conduct experiments on a multi-server test-bed to demonstrate that our algorithms can effectively save transmitted bits without performance degradation.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributed OptimizationQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DEED: A General Quantization Scheme for Communication Efficiency in Bits
In distributed optimization, a popular technique to reduce communication is quantization. In this paper, we provide a general analysis framework for inexact gradient descent that is applicable to quantization schemes. We…
Distributed OptimizationFederated LearningQuantizationQuantization Avoids Saddle Points in Distributed Optimization
Distributed nonconvex optimization underpins key functionalities of numerous distributed systems, ranging from power systems, smart buildings, cooperative robots, vehicle networks to sensor networks. Recently, it has als…
Distributed OptimizationQuantizationDistributed Optimization via Gradient Descent with Event-Triggered Zooming over Quantized Communication
In this paper, we study unconstrained distributed optimization strongly convex problems, in which the exchange of information in the network is captured by a directed graph topology over digital channels that have limite…
Distributed OptimizationQuantizationReal-Time Distributed Model Predictive Control with Limited Communication Data Rates
The application of distributed model predictive controllers (DMPC) for multi-agent systems (MASs) necessitates communication between agents, yet the consequence of communication data rates is typically overlooked. This w…
Distributed OptimizationModel Predictive ControlQuantizationFinite-Bit Quantization For Distributed Algorithms With Linear Convergence
This paper studies distributed algorithms for (strongly convex) composite optimization problems over mesh networks, subject to quantized communications. Instead of focusing on a specific algorithmic design, a black-box m…
Quantization