Adapting a Language Model While Preserving its General Knowledge
Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach.
Code (2)
Tasks
Continual LearningGeneral KnowledgeLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
Despite extensive research, open-vocabulary segmentation methods still struggle to generalize across diverse domains. To reduce the computational cost of adapting Vision-Language Models (VLMs) while preserving their pre-…
Language ModelingLanguage ModellingOpen Vocabulary Panoptic SegmentationPanoptic SegmentationAdapting Vision-Language Models for E-commerce Understanding at Scale
E-commerce product understanding demands by nature, strong multimodal comprehension from text, images, and structured attributes. General-purpose Vision-Language Models (VLMs) enable generalizable multimodal latent model…
Instruction FollowingAttribute ExtractionEvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation
The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge. While parameter-efficient prompt learning methods offer a promising path, they of…
Few-Shot LearningSelect and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of downstream tasks often leads to the forgetting …
Continual LearningKnowledge DistillationTransfer Learningzero-shot-classification+2Modular Multi-Task Learning for Chemical Reaction Prediction
Adapting large language models (LLMs) trained on broad organic chemistry to smaller, domain-specific reaction datasets is a key challenge in chemical and pharmaceutical R&D. Effective specialisation requires learning new…
parameter-efficient fine-tuningChemical Reaction PredictionForward reaction predictionMulti-Task Learning