DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization
Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.
Code (2)
Tasks
Knowledge DistillationModel CompressionQuantizationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. C…
DecoderKnowledge DistillationModel CompressionNeural Network Compression+2BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese
We present BARTpho with two versions, BARTpho-syllable and BARTpho-word, which are the first public large-scale monolingual sequence-to-sequence models pre-trained for Vietnamese. BARTpho uses the "large" architecture an…
Abstractive Text SummarizationDenoisingPunctuation RestorationText SummarizationEnhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study
Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages. Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated i…
Knowledge DistillationLanguage ModelingLanguage ModellingLow Resource NMT+3AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization
Like most natural language understanding and generation tasks, state-of-the-art models for summarization are transformer-based sequence-to-sequence architectures that are pretrained on large corpora. While most existing …
Abstractive Text SummarizationDecoderNatural Language UnderstandingAraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization
Like most natural language understanding and generation tasks, state-of-the-art models for summarization are transformer-based sequence-to-sequence architectures that are pretrained on large corpora. While most existing …
Abstractive Text SummarizationDecoderNatural Language Understanding