Fast Adjustable Threshold For Uniform Neural Network Quantization (Winning solution of LPIRC-II)
Neural network quantization procedure is the necessary step for porting of neural networks to mobile devices. Quantization allows accelerating the inference, reducing memory consumption and model size. It can be performed without fine-tuning using calibration procedure (calculation of parameters necessary for quantization), or it is possible to train the network with quantization from scratch. Training with quantization from scratch on the labeled data is rather long and resource-consuming procedure. Quantization of network without fine-tuning leads to accuracy drop because of outliers which appear during the calibration. In this article we suggest to simplify the quantization procedure significantly by introducing the trained scale factors for quantization thresholds. It allows speeding up the process of quantization with fine-tuning up to 8 epochs as well as reducing the requirements to the set of train images. By our knowledge, the proposed method allowed us to get the first public available quantized version of MNAS without significant accuracy reduction - 74.8% vs 75.3% for original full-precision network. Model and code are ready for use and available at: https://github.com/agoncharenko1992/FAT-fast_adjustable_threshold.
Code (4)
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonunifo…
Quantization$\text{Log}_\text{b}$Quant: Quantizing Language Models in Logarithmic Space
Quantization has become an invaluable tool to reduce memory requirements and inference speed of modern language models, in particular to make them available for consumer setups and edge devices. While previous work has p…
Received Power Maximization Using Nonuniform Discrete Phase Shifts for RISs With a Limited Phase Range
To maximize the received power at a user equipment, the problem of optimizing a reconfigurable intelligent surface (RIS) with a limited phase range R < 2{\pi} and nonuniform discrete phase shifts with adjustable gains is…
2kQuantizationRoPE-Aware Bit Allocation for KV-Cache Quantization
Existing low-bit KV-cache quantizers often treat each cached key as a flat vector. Under RoPE, however, a key's contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequ…
On Uniform Scalar Quantization for Learned Image Compression
Learned image compression possesses a unique challenge when incorporating non-differentiable quantization into the gradient-based training of the networks. Several quantization surrogates have been proposed to fulfill th…
Image CompressionQuantization