Automatic Domain Adaptation by Transformers in In-Context Learning

2024-05-27 · Ryuichiro Hataya, Kota Matsui, Masaaki Imaizumi

Selecting or designing an appropriate domain adaptation algorithm for a given problem remains challenging. This paper presents a Transformer model that can provably approximate and opt for domain adaptation methods for a given dataset in the in-context learning framework, where a foundation model performs new tasks without updating its parameters at test time. Specifically, we prove that Transformers can approximate instance-based and feature-based unsupervised domain adaptation algorithms and automatically select an algorithm suited for a given dataset. Numerical results indicate that in-context learning demonstrates an adaptive domain adaptation surpassing existing methods.

📄 PDF Abstract BibTeX arXiv:2405.16819

Code (0)

등록된 구현이 없습니다.

Tasks

Domain AdaptationIn-Context LearningUnsupervised Domain Adaptation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Multi-Head Attention 설명 없음

Crisis Domain Adaptation Using Sequence-to-sequence Transformers

2021-10-15 · Congcong Wang, Paul Nulty, David Lillis

User-generated content (UGC) on social media can act as a key source of information for emergency responders in crisis situations. However, due to the volume concerned, computational techniques are needed to effectively …

Domain Adaptation Language Modelling

Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes

2020-06-25 · ACL 2019 7 · Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze

In this paper, we automatically create sentiment dictionaries for predicting financial outcomes. We compare three approaches: (I) manual adaptation of the domain-general dictionary H4N, (ii) automatic adaptation of H4N a…

Domain Adaptation

QA Domain Adaptation using Data Augmentation and Contrastive Adaptation

2021-11-16 · ACL ARR November 2021 11 · Anonymous

Domain adaptation for question answering (QA) has recently shown impressive results for answering out-of-domain questions. Yet, a common challenge is to build approaches that are effective for niche domains with small te…

Data Augmentation Domain Adaptation Question Answering Question Generation+1

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers

2025-09-12 · Akshat Pandey, Karun Kumar, Raphael Tang arxiv

Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In many real-world settings, collecting speech data is impractical, necessitat…

Speech Recognition Domain Adaptation

Translation Transformers Rediscover Inherent Data Domains

2021-09-16 · WMT (EMNLP) 2021 11 · Maksym Del, Elizaveta Korotkova, Mark Fishel

Many works proposed methods to improve the performance of Neural Machine Translation (NMT) models in a domain/multi-domain adaptation scenario. However, an understanding of how NMT baselines represent text domain informa…

Clustering Domain Adaptation Machine Translation NMT+3