UDApter -- Efficient Domain Adaptation Using Adapters
We propose two methods to make unsupervised domain adaptation (UDA) more parameter efficient using adapters, small bottleneck layers interspersed with every layer of the large-scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain-invariant information to learn task representations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Compared to strong baselines, our simple methods perform well in natural language inference (MNLI) and the cross-domain sentiment classification task. We even outperform unsupervised domain adaptation methods such as DANN and DSN in sentiment classification, and we are within 0.85% F1 for natural language inference task, by fine-tuning only a fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter
Code (1)
Tasks
Domain AdaptationLanguage ModelingLanguage ModellingNatural Language InferenceSentiment AnalysisSentiment ClassificationUnsupervised Domain AdaptationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UDapter: Language Adaptation for Truly Universal Dependency Parsing
Recent advances in multilingual dependency parsing have brought the idea of a truly universal parser closer to reality. However, cross-language interference and restrained model capacity remain major obstacles. To addres…
Dependency ParsingTransfer LearningUDapter: Typology-based Language Adapters for Multilingual Dependency Parsing and Sequence Labeling
Recent advances in multilingual language modeling have brought the idea of a truly universal parser closer to reality. However, such models are still not immune to the “curse of multilinguality”: Cross-language interfere…
Dependency ParsingLanguage ModelingLanguage ModellingMorphological Tagging+3MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation
Pre-trained language models struggle when applied to new domains, as full fine-tuning is computationally expensive and prone to catastrophic forgetting. This study addresses this challenge by presenting a novel parameter…
Unsupervised Domain AdaptationNatural Language InferenceMultilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural machine translation (NMT), to adapt pre-trained models to new domains or language pai…
Cross-Lingual TransferDomain AdaptationMachine TranslationNMT+1EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference
The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuni…