NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes gradients to reduce communication costs. The baseline variant of QSGD provides strong theoretical guarantees, however, for practical purposes, the authors proposed a heuristic variant which we call QSGDinf, which demonstrated impressive empirical gains for distributed training of large neural networks. In this paper, we build on this work to propose a new gradient quantization scheme, and show that it has both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical performance of the QSGDinf heuristic and of other compression methods.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NUQSGD: Improved Communication Efficiency for Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel. Alistarh…
QuantizationProvably Communication-efficient Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel. Alistarh…
QuantizationState Synchronization of Discrete-time Multi-agent Systems in Presence of Unknown Nonuniform Communication Delays: A Scale-free Protocol Design
In this paper, we study scale-free state synchronization of discrete-time homogeneous multi-agent systems (MAS) subject to unknown, nonuniform, and arbitrarily large communication delays. The scale-free protocol utilizes…
Scalable tensor methods for nonuniform hypergraphs
While multilinear algebra appears natural for studying the multiway interactions modeled by hypergraphs, tensor methods for general hypergraphs have been stymied by theoretical and practical barriers. A recently proposed…
Parallel framework for Dynamic Domain Decomposition of Data Assimilation problems a case study on Kalman Filter algorithm
We focus on Partial Differential Equation (PDE) based Data Assimilatio problems (DA) solved by means of variational approaches and Kalman filter algorithm. Recently, we presented a Domain Decomposition framework (we call…
State Estimation