Serial Contrastive Knowledge Distillation for Continual Few-shot Relation Extraction
Continual few-shot relation extraction (RE) aims to continuously train a model for new relations with few labeled training data, of which the major challenges are the catastrophic forgetting of old relations and the overfitting caused by data sparsity. In this paper, we propose a new model, namely SCKD, to accomplish the continual few-shot RE task. Specifically, we design serial knowledge distillation to preserve the prior knowledge from previous models and conduct contrastive learning with pseudo samples to keep the representations of samples in different relations sufficiently distinguishable. Our experiments on two benchmark datasets validate the effectiveness of SCKD for continual few-shot RE and its superiority in knowledge transfer and memory utilization over state-of-the-art models.
Code (1)
Tasks
Contrastive LearningKnowledge DistillationRelationRelation ExtractionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Provable Contrastive Continual Learning
Continual learning requires learning incremental tasks with dynamic data distributions. So far, it has been observed that employing a combination of contrastive loss and distillation loss for training in continual learni…
Continual LearningContrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation
Recently, because of the high-quality representations of contrastive learning methods, rehearsal-based contrastive continual learning has been proposed to explore how to continually learn transferable representation embe…
Continual LearningContrastive LearningRelationRepresentation LearningA Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation
Most fine-grained fashion image retrieval (FIR) methods assume a static setting, requiring full retraining when new attributes appear, which is costly and impractical for dynamic scenarios. Although pretrained models sup…
class-incremental learningContrastive LearningContinual LearningImage RetrievalCLIP model is an Efficient Continual Learner
The continual learning setting aims to learn new tasks over time without forgetting the previous ones. The literature reports several significant efforts to tackle this problem with limited or no access to previous task …
Continual LearningIncremental LearningKnowledge DistillationmodelCLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks
This paper studies continual learning (CL) of a sequence of aspect sentiment classification(ASC) tasks in a particular CL setting called domain incremental learning (DIL). Each task is from a different domain or product.…
ClassificationContinual LearningContrastive LearningIncremental Learning+4