Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
In this work, we quantize a trained Transformer machine language translation model leveraging INT8/VNNI instructions in the latest Intel$^\circledR$ Xeon$^\circledR$ Cascade Lake processors to improve inference performance while maintaining less than 0.5$\%$ drop in accuracy. To the best of our knowledge, this is the first attempt in the industry to quantize the Transformer model. This has high impact as it clearly demonstrates the various complexities of quantizing the language translation model. We present novel quantization techniques directly in TensorFlow to opportunistically replace 32-bit floating point (FP32) computations with 8-bit integers (INT8) and transform the FP32 computational graph. We also present a bin-packing parallel batching technique to maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver 1.5X improvement over the best FP32 performance. Furthermore, it reveals the opportunities and challenges to boost performance of quantized deep learning inference and establishes best practices to run inference with high efficiency on Intel CPUs.
Code (0)
등록된 구현이 없습니다.
Tasks
CPUQuantizationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fully Quantized Transformer for Machine Translation
State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsuccessful. To …
Machine TranslationQuantizationTranslationNeural Machine Translation with 4-Bit Precision and Beyond
Neural Machine Translation (NMT) is resource intensive. We design a quantization procedure to compress NMT models better for devices with limited hardware capability. Because most neural network parameters are near zero,…
Machine TranslationNMTQuantizationTranslationThe Uneven Impact of Post-Training Quantization in Machine Translation
Quantization is essential for deploying large language models (LLMs) on resource-constrained hardware, but its implications for multilingual tasks remain underexplored. We conduct the first large-scale evaluation of post…
Machine TranslationExtremely Low Bit Transformer Quantization for On-Device Neural Machine Translation
The deployment of widely used Transformer architecture is challenging because of heavy computation load and memory overhead during inference, especially when the target device is limited in computational resources such a…
Machine TranslationNMTQuantizationTranslationBinarized Neural Machine Translation
The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its…
BinarizationMachine TranslationQuantizationTranslation