N-Ary Quantization for CNN Model Compression and Inference Acceleration
The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringModel CompressionQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference
With the rapid growth in the scale and complexity of large language models (LLMs), the costs of training and inference have risen substantially. Model compression has emerged as a mainstream solution to reduce memory usa…
GPULanguage ModelingLanguage ModellingLarge Language Model+2ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
Large Language Models (LLMs) have revolutionized natural language processing tasks. However, their practical application is constrained by substantial memory and computational demands. Post-training quantization (PTQ) is…
GPUModel CompressionQuantizationOn the Expressive Power of Weight Quantization in Large Language Models
In recent years, weight quantization that encodes the learnable parameters of large language models in an $n$-bit format has garnered significant attention due to its potential for model compression and inference acceler…
Model CompressionHACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
Disaggregated Large Language Model (LLM) inference has gained popularity as it separates the computation-intensive prefill stage from the memory-intensive decode stage, avoiding the prefill-decode interference and improv…
Language ModelingLanguage ModellingLarge Language ModelQuantizationModel Compression and Efficient Inference for Large Language Models: A Survey
Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resour…
Knowledge DistillationModel CompressionQuantization