CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers
Contrastive Language-Image Pre-training (CLIP) has been shown to improve zero-shot generalization capabilities of language and vision models. In this paper, we extend CLIP for efficient knowledge distillation, by utilizing embeddings as teachers. Typical knowledge distillation frameworks require running forward passes through a teacher model, which is often prohibitive in the case of billion or trillion parameter teachers. In these cases, using only the embeddings of the teacher models to guide the distillation can yield significant computational savings. Our preliminary findings show that CLIP-based knowledge distillation with embeddings can outperform full scale knowledge distillation using $9\times$ less memory and $8\times$ less training time. Code available at: https://github.com/lnairGT/CLIP-Distillation/
Code (1)
Tasks
Knowledge DistillationZero-shot GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CLIP-RD: Relative Distillation for Efficient CLIP Knowledge Distillation
CLIP aligns image and text embeddings via contrastive learning and demonstrates strong zero-shot generalization. Its large-scale architecture requires substantial computational and memory resources, motivating the distil…
Zero-shot GeneralizationKnowledge DistillationContrastive LearningEnhancing CLIP Conceptual Embedding through Knowledge Distillation
Recently, CLIP has become an important model for aligning images and text in multi-modal contexts. However, researchers have identified limitations in the ability of CLIP's text and image encoders to extract detailed kno…
Contrastive LearningKnowledge DistillationWords Matter: Leveraging Individual Text Embeddings for Code Generation in CLIP Test-Time Adaptation
Vision-language foundation models, such as CLIP, have shown unprecedented zero-shot performance across a wide range of tasks. Nevertheless, these models may be unreliable under distributional shifts, as their performance…
Code GenerationContrastive LearningKnowledge DistillationRepresentation Learning+1MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
We present a new method of self-supervised learning and knowledge distillation based on the multi-views and multi-representations (MV-MR). The MV-MR is based on the maximization of dependence between learnable embeddings…
ClusteringContrastive LearningKnowledge DistillationLinear evaluation+3Audio Embeddings as Teachers for Music Classification
Music classification has been one of the most popular tasks in the field of music information retrieval. With the development of deep learning models, the last decade has seen impressive improvements in a wide range of c…
ClassificationInformation RetrievalInstrument RecognitionKnowledge Distillation+5