Analysis of Quantized Models
Weight-quantized networks have small storage and fast inference, but training can still be time-consuming. This can be improved with distributed learning. To reduce the high communication cost due to worker-server synchronization, recently gradient quantization has also been proposed to train networks with full-precision weights. In this paper, we theoretically study how the combination of both weight and gradient quantization affects convergence. We show that (i) weight-quantized networks converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization. Empirical experiments confirm the theoretical convergence results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Quantized Fisher Discriminant Analysis
This paper proposes a new subspace learning method, named Quantized Fisher Discriminant Analysis (QFDA), which makes use of both machine learning and information theory. There is a lack of literature for combination of m…
BIG-bench Machine LearningDimensionality ReductionQuantizationLinear and Nonlinear MMSE Estimation in One-Bit Quantized Systems under a Gaussian Mixture Prior
We present new fundamental results for the mean square error (MSE)-optimal conditional mean estimator (CME) in one-bit quantized systems for a Gaussian mixture model (GMM) distributed signal of interest, possibly corrupt…
QuantizationA Unified Framework for Quantized and Continuous Strong Lottery Tickets
The Strong Lottery Ticket Hypothesis (SLTH) asserts that sufficiently overparameterized, randomly initialized neural networks contain sparse subnetworks that, even without any training, can match the performance of a sma…
Quantized FCA: Efficient Zero-Shot Texture Anomaly Detection
Zero-shot anomaly localization is a rising field in computer vision research, with important progress in recent years. This work focuses on the problem of detecting and localizing anomalies in textures, where anomalies c…
Anomaly DetectionStability and $\mathcal{H}_{\infty}$ Performance Analysis of Stochastic Linear Networked and Quantized Control Systems
This paper studies the stability and $\mathcal{H}_{\infty}$ performance analysis problem for linear networked and quantized control systems with both communication delays random packet losses. To deal with the network-in…