Q8BERT: Quantized 8Bit BERT
Recently, pre-trained Transformer based language models such as BERT and GPT, have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 and Megatron, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by $4\times$ with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.
Code (5)
Tasks
Linguistic AcceptabilityNatural Language InferenceQuantizationSemantic Textual SimilaritySentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Effective and Efficient Mixed Precision Quantization of Speech Foundation Models
This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estimation into one single model compression s…
Model Compressionparameter estimationQuantizationHardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing
BERT is the most recent Transformer-based model that achieves state-of-the-art performance in various NLP tasks. In this paper, we investigate the hardware acceleration of BERT on FPGA for edge computing. To tackle the i…
CPUEdge-computingGPUQuaLA-MiniLM: a Quantized Length Adaptive MiniLM
Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distil…
Computational EfficiencyKnowledge DistillationQuantizationQuestion AnsweringQ-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation
Sequential recommendation plays a critical role in modern online platforms such as e-commerce, advertising, and content streaming, where accurately predicting users' next interactions is essential for personalization. Re…
Sequential RecommendationMultimodal RecommendationRepresentation LearningMKQ-BERT: Quantized BERT with 4-bits Weights and Activations
Recently, pre-trained Transformer based language models, such as BERT, have shown great superiority over the traditional methods in many Natural Language Processing (NLP) tasks. However, the computational cost for deploy…
Quantization