Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel. Alistarh et al. (2017) describe two variants of data-parallel SGD that quantize and encode gradients to lessen communication costs. For the first variant, QSGD, they provide strong theoretical guarantees. For the second variant, which we call QSGDinf, they demonstrate impressive empirical gains for distributed training of large neural networks. Building on their work, we propose an alternative scheme for quantizing gradients and show that it yields stronger theoretical guarantees than exist for QSGD while matching the empirical performance of QSGDinf.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSimilar Papers 제목 키워드 기반
NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communicat…
QuantizationNUQSGD: Improved Communication Efficiency for Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel. Alistarh…
QuantizationNonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonunifo…
QuantizationIntra-layer Nonuniform Quantization for Deep Convolutional Neural Network
Deep convolutional neural network (DCNN) has achieved remarkable performance on object detection and speech recognition in recent years. However, the excellent performance of a DCNN incurs high computational complexity a…
ClusteringGeneral Classificationobject-detectionObject Detection+3Received Power Maximization Using Nonuniform Discrete Phase Shifts for RISs With a Limited Phase Range
To maximize the received power at a user equipment, the problem of optimizing a reconfigurable intelligent surface (RIS) with a limited phase range R < 2{\pi} and nonuniform discrete phase shifts with adjustable gains is…
2kQuantization