Multi-perspective Contrastive Logit Distillation
We propose a novel and efficient logit distillation method, Multi-perspective Contrastive Logit Distillation (MCLD), which leverages contrastive learning to distill logits from multiple perspectives in knowledge distillation. Recent research on logit distillation has primarily focused on maximizing the information learned from the teacher model's logits to enhance the performance of the student model. To this end, we propose MCLD, which consists of three key components: Instance-wise CLD, Sample-wise CLD, and Category-wise CLD. These components are designed to facilitate the transfer of more information from the teacher's logits to the student model. Comprehensive evaluations on image classification tasks using CIFAR-100 and ImageNet, alongside representation transferability assessments on STL-10 and Tiny-ImageNet, highlight the significant advantages of our method. The knowledge distillation with our MCLD, surpasses existing state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive Learningimage-classificationImage ClassificationKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective
In this paper, we present a simple yet effective contrastive knowledge distillation approach, which can be formulated as a sample-wise alignment problem with intra- and inter-sample constraints. Unlike traditional knowle…
Contrastive Learningimage-classificationImage ClassificationKnowledge Distillation+2Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level super…
Reinforcement LearningContrastive LearningDecoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
Knowledge Distillation (KD), a learning manner with a larger teacher network guiding a smaller student network, transfers dark knowledge from the teacher to the student via logits or intermediate features, with the aim o…
Knowledge DistillationPhilosophyDistillCSE: Distilled Contrastive Learning for Sentence Embeddings
This paper proposes the DistillCSE framework, which performs contrastive learning under the self-training paradigm with knowledge distillation. The potential advantage of DistillCSE is its self-enhancing feature: using a…
Contrastive LearningKnowledge DistillationSentenceSentence EmbeddingsRethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective
In the history of knowledge distillation, the focus has once shifted over time from logit-based to feature-based approaches. However, this transition has been revisited with the advent of Decoupled Knowledge Distillation…
Knowledge Distillation