UDALM: Unsupervised Domain Adaptation through Language Modeling

2021-04-14 · NAACL 2021 4 · Constantinos Karouzos, Georgios Paraskevopoulos, Alexandros Potamianos

In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding $91.74\%$ accuracy, which is an $1.11\%$ absolute improvement over the state-of-the-art.

📄 PDF Abstract BibTeX arXiv:2104.07078

Code (1)

ckarouzos/slp_daptmlm 공식 구현 pytorch

Tasks

Domain AdaptationLanguage ModelingLanguage Modelling Sentiment Analysis Unsupervised Domain Adaptation

MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR

2025-05-30 · Dimitrios Damianos, Georgios Paraskevopoulos, Alexandros Potamianos

In this work, we investigate the Meta PL unsupervised domain adaptation framework for Automatic Speech Recognition (ASR). We introduce a Multi-Stage Domain Adaptation pipeline (MSDA), a sample-efficient, two-stage adapta…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Domain Adaptation Self-Supervised Learning+3

MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation

2026-06-20 · Mohammed Rawhani, Dervis Karaboga, Ozkan Ufuk Nalbantoglu, Alper Basturk 외 arxiv

Pre-trained language models struggle when applied to new domains, as full fine-tuning is computationally expensive and prone to catastrophic forgetting. This study addresses this challenge by presenting a novel parameter…

Unsupervised Domain Adaptation Natural Language Inference

Model Selection with Nonlinear Embedding for Unsupervised Domain Adaptation

2017-06-23 · Hemanth Venkateswara, Shayok Chakraborty, Troy McDaniel, Sethuraman Panchanathan

Domain adaptation deals with adapting classifiers trained on data from a source distribution, to work effectively on data from a target distribution. In this paper, we introduce the Nonlinear Embedding Transform (NET) fo…

Domain Adaptation General Classification Model Selection Unsupervised Domain Adaptation

Knowledge Distillation for BERT Unsupervised Domain Adaptation

2020-10-22 · Minho Ryu, Kichun Lee

A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. Since the model is trained on a large corpus of diverse topics, it shows robust pe…

Domain Adaptation General Classification Knowledge Distillation Language Modeling+4

Unsupervised Domain Adaptation Through Transferring both the Source-Knowledge and Target-Relatedness Simultaneously

2020-03-18 · Qing Tian, Yanan Zhu, Chuang Ma, Meng Cao

Unsupervised domain adaptation (UDA) is an emerging research topic in the field of machine learning and pattern recognition, which aims to help the learning of unlabeled target domain by transferring knowledge from the s…

BIG-bench Machine Learning Domain Adaptation Unsupervised Domain Adaptation