Pruning vs Quantization: Which is Better?
Neural network pruning and quantization techniques are almost as old as neural networks themselves. However, to date only ad-hoc comparisons between the two have been published. In this paper, we set out to answer the question on which is better: neural network quantization or pruning? By answering this question, we hope to inform design decisions made on neural network hardware going forward. We provide an extensive comparison between the two techniques for compressing deep neural networks. First, we give an analytical comparison of expected quantization and pruning error for general data distributions. Then, we provide lower bounds for the per-layer pruning and quantization error in trained networks, and compare these to empirical error after optimization. Finally, we provide an extensive experimental comparison for training 8 large-scale models on 3 tasks. Our results show that in most cases quantization outperforms pruning. Only in some scenarios with very high compression ratio, pruning might be beneficial from an accuracy standpoint.
Code (1)
Tasks
Network PruningQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Differentiable Joint Pruning and Quantization for Hardware Efficiency
We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatical…
Neural Network CompressionQuantizationPs and Qs: Quantization-aware pruning for efficient low latency neural network inference
Efficient machine learning implementations optimized for inference in hardware have wide-ranging benefits, depending on the application, from lower inference latency to higher data throughput and reduced energy consumpti…
Bayesian OptimizationComputational EfficiencyNeural Architecture SearchQuantizationAutomated Model Compression by Jointly Applied Pruning and Quantization
In the traditional deep compression framework, iteratively performing network pruning and quantization can reduce the model size and computation cost to meet the deployment requirements. However, such a step-wise applica…
AutoMLModel CompressionNetwork Pruningobject-detection+2Bayesian Bits: Unifying Quantization and Pruning
We introduce Bayesian Bits, a practical method for joint mixed precision quantization and pruning through gradient based optimization. Bayesian Bits employs a novel decomposition of the quantization operation, which sequ…
QuantizationNeural Network Compression using Binarization and Few Full-Precision Weights
Quantization and pruning are two effective Deep Neural Networks model compression methods. In this paper, we propose Automatic Prune Binarization (APB), a novel compression technique combining quantization with pruning. …
BinarizationCPUModel CompressionNeural Network Compression+1