Towards Superior Quantization Accuracy: A Layer-sensitive Approach
Large Vision and Language Models have exhibited remarkable human-like intelligence in tasks such as natural language comprehension, problem-solving, logical reasoning, and knowledge retrieval. However, training and serving these models require substantial computational resources, posing a significant barrier to their widespread application and further research. To mitigate this challenge, various model compression techniques have been developed to reduce computational requirements. Nevertheless, existing methods often employ uniform quantization configurations, failing to account for the varying difficulties across different layers in quantizing large neural network models. This paper tackles this issue by leveraging layer-sensitivity features, such as activation sensitivity and weight distribution Kurtosis, to identify layers that are challenging to quantize accurately and allocate additional memory budget. The proposed methods, named SensiBoost and KurtBoost, respectively, demonstrate notable improvement in quantization accuracy, achieving up to 9% lower perplexity with only a 2% increase in memory budget on LLama models compared to the baseline.
Code (0)
등록된 구현이 없습니다.
Tasks
Logical ReasoningModel CompressionQuantizationSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Automatic Network Adaptation for Ultra-Low Uniform-Precision Quantization
Uniform-precision neural network quantization has gained popularity since it simplifies densely packed arithmetic unit for high computing capability. However, it ignores heterogeneous sensitivity to the impact of quantiz…
Neural Architecture SearchQuantizationUniform-Precision Neural Network Quantization via Neural Channel Expansion
Uniform-precision neural network quantization has gained popularity thanks to its simple arithmetic unit densely packed for high computing capability. However, it ignores heterogeneous sensitivity to the impact of quanti…
Neural Architecture SearchQuantizationSliderQuant: Accurate Post-Training Quantization for LLMs
In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats…
Towards Effective 2-bit Quantization: Pareto-optimal Bit Allocation for Deep CNNs Compression
State-of-the-art quantization methods can compress deep neural networks down to 4 bits without losing accuracy. However, when it comes to 2 bits, the performance drop is still noticeable. One problem in these methods is …
QuantizationAnalysis of Quantization on MLP-based Vision Models
Quantization is wildly taken as a model compression technique, which obtains efficient models by converting floating-point weights and activations in the neural network into lower-bit integers. Quantization has been prov…
Model CompressionQuantization