Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning
Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the \emph{Semantically-guided Representation Learning (SG-RL)} module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning.
Code (1)
Tasks
Continual LearningKnowledge DistillationRepresentation LearningSemantic SimilaritySemantic Textual SimilarityTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fine-tuning Multimodal Large Language Models for Product Bundling
Recent advances in product bundling have leveraged multimodal information through sophisticated encoders, but remain constrained by limited semantic understanding and a narrow scope of knowledge. Therefore, some attempts…
In-Context LearningMultiple-choiceTransferring Semantic Knowledge Into Language Encoders
We introduce semantic form mid-tuning, an approach for transferring semantic knowledge from semantic meaning representations into transformer-based language encoders. In mid-tuning, we learn to align the text of general …
Reading ComprehensionSentenceSentence EmbeddingSentence-Embedding+1On the impressive performance of randomly weighted encoders in summarization tasks
In this work, we investigate the performance of untrained randomly initialized encoders in a general class of sequence to sequence models and compare their performance with that of fully-trained encoders on the task of a…
Abstractive Text SummarizationDecoderTransformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence
This paper proposes a transformer over transformer framework, called Transformer$^2$, to perform neural text segmentation. It consists of two components: bottom-level sentence encoders using pre-trained transformers, and…
SegmentationSentenceSentence EmbeddingsText SegmentationLexFit: Lexical Fine-Tuning of Pretrained Language Models
Transformer-based language models (LMs) pretrained on large text collections implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. Ins…
Cross-Lingual Transfer