Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation
Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both the general-domain and in-domain. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, domain divergence, and model explosion. To address these three problems, we propose a method of "divide and conquer" which is based on the importance of neurons or parameters in the translation model. In our method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original unpruned model with the knowledge distillation method. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conduct experiments on different languages and domains and the results show that our method can achieve significant improvements compared with several strong baselines.
Code (1)
Tasks
Domain AdaptationKnowledge DistillationMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension
In this paper, we propose a simple few-shot domain adaptation paradigm for reading comprehension. We first identify the lottery subnetwork structure within the Transformer-based source domain model via gradual magnitude …
Domain AdaptationReading ComprehensionAll-in-One Tuning and Structural Pruning for Domain-Specific LLMs
Existing pruning techniques for large language models (LLMs) targeting domain-specific applications typically follow a two-stage process: pruning the pretrained general-purpose LLMs and then fine-tuning the pruned LLMs o…
AllPADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations
While self-supervised speech representation learning (SSL) models serve a variety of downstream tasks, these models have been observed to overfit to the domain from which the unlabelled data originates. To alleviate this…
Domain AdaptationLanguage ModellingRepresentation LearningSpeech Representation LearningUnsupervised Domain Adaptation Via Data Pruning
The removal of carefully-selected examples from training data has recently emerged as an effective way of improving the robustness of machine learning models. However, the best way to select these examples remains an ope…
Domain AdaptationEvent DetectionUnsupervised Domain AdaptationMonolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language,…
Instruction FollowingLarge Language ModelMultilingual NLPOpen-Domain Question Answering+2