Robustly Optimized and Distilled Training for Natural Language Understanding
In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise a KD variant that learns effectively from multiple teachers. By combining MTL and KD, we propose Robustly Optimized and Distilled (ROaD) modeling framework. We use ROaD together with the ELECTRA model to obtain state-of-the-art results for machine reading comprehension and natural language inference.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine Reading ComprehensionMulti-Task LearningNatural Language InferenceNatural Language UnderstandingReading ComprehensionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Transformer-based approaches to Sentiment Detection
The use of transfer learning methods is largely responsible for the present breakthrough in Natural Learning Processing (NLP) tasks across multiple domains. In order to solve the problem of sentiment detection, we examin…
text-classificationText ClassificationTransfer LearningLeveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies
Emerging Large Language Models (LLMs) like GPT-4 have revolutionized Natural Language Processing (NLP), showing potential in traditional tasks such as Named Entity Recognition (NER). Our study explores a three-phase trai…
Knowledge Distillationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU…
Language ModelingLanguage ModellingLinguistic AcceptabilityNatural Language Inference+7Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughpu…
MambaAlexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, …
Cross-Lingual Natural Language Inferenceintent-classificationIntent ClassificationNatural Language Inference+4