Complementary Relation Contrastive Distillation
Knowledge distillation aims to transfer representation ability from a teacher model to a student model. Previous approaches focus on either individual representation distillation or inter-sample similarity preservation. While we argue that the inter-sample relation conveys abundant information and needs to be distilled in a more effective way. In this paper, we propose a novel knowledge distillation method, namely Complementary Relation Contrastive Distillation (CRCD), to transfer the structural knowledge from the teacher to the student. Specifically, we estimate the mutual relation in an anchor-based way and distill the anchor-student relation under the supervision of its corresponding anchor-teacher relation. To make it more robust, mutual relations are modeled by two complementary elements: the feature and its gradient. Furthermore, the low bound of mutual information between the anchor-teacher relation distribution and the anchor-student relation distribution is maximized via relation contrastive loss, which can distill both the sample representation and the inter-sample relations. Experiments on different benchmarks demonstrate the effectiveness of our proposed CRCD.
Code (1)
Tasks
Knowledge DistillationRelationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Complementary Calibration: Boosting General Continual Learning with Collaborative Distillation and Self-Supervision
General Continual Learning (GCL) aims at learning from non independent and identically distributed stream data without catastrophic forgetting of the old tasks that don't rely on task boundaries during both training and …
Continual LearningContrastive LearningKnowledge DistillationRelationDiscriminative and Consistent Representation Distillation
Knowledge Distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model. While contrastive learning has shown promise in self-supervised learning by creating discriminative representa…
Causal InferenceContrastive LearningKnowledge DistillationSelf-Supervised LearningMaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. The core idea of masked self-distillation is to dis…
Representation LearningContrastive Continual Multi-view Clustering with Filtered Structural Fusion
Multi-view clustering thrives in applications where views are collected in advance by extracting consistent and complementary information among views. However, it overlooks scenarios where data views are collected sequen…
ClusteringContrastive LearningKnowledge DistillationKDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder
In this work, we attempted to extend the thought and showcase a way forward for the Self-supervised Learning (SSL) learning paradigm by combining contrastive learning, self-distillation (knowledge distillation) and maske…
Contrastive LearningKnowledge DistillationSelf-Supervised Learning