Data-free mixed-precision quantization using novel sensitivity metric
Post-training quantization is a representative technique for compressing neural networks, making them smaller and more efficient for deployment on edge devices. However, an inaccessible user dataset often makes it difficult to ensure the quality of the quantized neural network in practice. In addition, existing approaches may use a single uniform bit-width across the network, resulting in significant accuracy degradation at extremely low bit-widths. To utilize multiple bit-width, sensitivity metric plays a key role in balancing accuracy and compression. In this paper, we propose a novel sensitivity metric that considers the effect of quantization error on task loss and interaction with other layers. Moreover, we develop labeled data generation methods that are not dependent on a specific operation of the neural network. Our experiments show that the proposed metric better represents quantization sensitivity, and generated data are more feasible to be applied to mixed-precision quantization.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSensitivitySimilar Papers 제목 키워드 기반
A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization
Recent advancement in Automatic Speech Recognition (ASR) has produced large AI models, which become impractical for deployment in mobile devices. Model quantization is effective to produce compressed general-purpose mode…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Quantizationspeech-recognition+1Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing methods typically treat all intra-layer we…
Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Low-bit neur…
Neural Architecture SearchQuantizationSensitivityspeech-recognition+1Patch-wise Mixed-Precision Quantization of Vision Transformer
As emerging hardware begins to support mixed bit-width arithmetic computation, mixed-precision quantization is widely used to reduce the complexity of neural networks. However, Vision Transformers (ViTs) require complex …
QuantizationSensitivityMixed Precision of Quantization of Transformer Language Models for Speech Recognition
State-of-the-art neural language models represented by Transformers are becoming increasingly complex and expensive for practical applications. Low-bit deep neural network quantization techniques provides a powerful solu…
QuantizationSensitivityspeech-recognitionSpeech Recognition