Knowledge Distillation for BERT Unsupervised Domain Adaptation
A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. Since the model is trained on a large corpus of diverse topics, it shows robust performance for domain shift problems in which data distributions at training (source data) and testing (target data) differ while sharing similarities. Despite its great improvements compared to previous models, it still suffers from performance degradation due to domain shifts. To mitigate such problems, we propose a simple but effective unsupervised domain adaptation method, adversarial adaptation with distillation (AAD), which combines the adversarial discriminative domain adaptation (ADDA) framework with knowledge distillation. We evaluate our approach in the task of cross-domain sentiment classification on 30 domain pairs, advancing the state-of-the-art performance for unsupervised domain adaptation in text sentiment classification.
Code (1)
Tasks
Domain AdaptationGeneral ClassificationKnowledge DistillationLanguage ModelingLanguage ModellingSentiment AnalysisSentiment ClassificationUnsupervised Domain AdaptationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation
Conventional unsupervised multi-source domain adaptation (UMDA) methods assume all source domains can be accessed directly. This neglects the privacy-preserving policy, that is, all the data and computations must be kept…
Domain AdaptationKnowledge DistillationMulti-Source Unsupervised Domain AdaptationPrivacy Preserving+1Unsupervised Domain Adaptation of Contextual Embeddings for Low-Resource Duplicate Question Detection
Answering questions is a primary goal of many conversational systems or search products. While most current systems have focused on answering questions against structured databases or curated knowledge graphs, on-line co…
Domain AdaptationKnowledge GraphsQuestion AnsweringSelf-Supervised Learning+1Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation
Unsupervised Domain Adaptation (UDA) aims to transfer the knowledge of source domain to the unlabeled target domain. Existing methods typically require to learn to adapt the target model by exploiting the source data and…
Cross-Domain Text ClassificationDomain AdaptationKnowledge Distillationtext-classification+2Domain Adaptation-Based Crossmodal Knowledge Distillation for 3D Semantic Segmentation
Semantic segmentation of 3D LiDAR data plays a pivotal role in autonomous driving. Traditional approaches rely on extensive annotated data for point cloud analysis, incurring high costs and time investments. In contrast,…
Unsupervised Domain Adaptation3D Semantic SegmentationKnowledge DistillationAutonomous DrivingLadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT t…
BlockingKnowledge DistillationModel CompressionNatural Language Understanding