Knowledge Distillation as Self-Supervised Learning
Self-supervised learning (SSL) methods have been shown to effectively train large neural networks with unlabeled data. These networks can produce useful image representations that can exceed the performance of supervised pretraining on downstream tasks. However, SSL is not effective with smaller models. This limits applications where computational power is limited, such as edge devices. Knowledge distillation (KD) is a popular method to train a smaller student network from a larger and more powerful teacher network. The SEED paper by Fang et al., published in ICLR 2021, applies knowledge distillation to self-supervised learning to pretrain smaller neural networks without supervision. In this post, we will discuss self-supervised learning and knowledge distillation and how they are unified in SEED.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
Self-supervised learning (SSL) has made remarkable progress in visual representation learning. Some studies combine SSL with knowledge distillation (SSL-KD) to boost the representation learning performance of small model…
Knowledge DistillationRepresentation LearningSelf-Supervised LearningAttention Distillation: self-supervised vision transformer students need more guidance
Self-supervised learning has been widely applied to train high-quality vision transformers. Unleashing their excellent performance on memory and compute constraint devices is therefore an important research topic. Howeve…
Knowledge DistillationSelf-Supervised LearningPrototypical Contrastive Predictive Coding
Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model to a smaller s…
Contrastive LearningKnowledge DistillationModel CompressionSelf-Supervised LearningDomain-Agnostic Clustering with Self-Distillation
Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmenta…
ClusteringData AugmentationDeep ClusteringKnowledge Distillation+2KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo
Supervised multi-view stereo (MVS) methods have achieved remarkable progress in terms of reconstruction quality, but suffer from the challenge of collecting large-scale ground-truth depth. In this paper, we propose a nov…
Knowledge DistillationSelf-Supervised Learning