Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
Distilling knowledge from huge pre-trained networks to improve the performance of tiny networks has favored deep learning models to be used in many real-time and mobile applications. Several approaches that demonstrate success in this field have made use of the true training dataset to extract relevant knowledge. In absence of the True dataset, however, extracting knowledge from deep networks is still a challenge. Recent works on data-free knowledge distillation demonstrate such techniques on classification tasks. To this end, we explore the task of data-free knowledge distillation for segmentation tasks. First, we identify several challenges specific to segmentation. We make use of the DeGAN training framework to propose a novel loss function for enforcing diversity in a setting where a few classes are underrepresented. Further, we explore a new training framework for performing knowledge distillation in a data-free setting. We get an improvement of 6.93% in Mean IoU over previous approaches.
Code (1)
Tasks
Data-free Knowledge DistillationDiversityKnowledge DistillationSegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Seeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation
Semantic segmentation requires a holistic understanding of the physical world, as it assigns semantic labels to spatially continuous and structurally coherent objects rather than to isolated pixels. However, existing dat…
Data-free Knowledge DistillationSemantic SegmentationSmall Scale Data-Free Knowledge Distillation
Data-free knowledge distillation is able to utilize the knowledge learned by a large teacher network to augment the training of a smaller student network without accessing the original training data, avoiding privacy, se…
Data-free Knowledge DistillationGenerative Adversarial Networkimage-classificationImage Classification+2Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less atten…
Contrastive LearningData Augmentationimage-classificationImage Classification+5Augmentation-free Dense Contrastive Distillation for Efficient Semantic Segmentation
In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less atten…
Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation
Source-free domain adaptation aims to adapt deep neural networks using only pre-trained source models and target data. However, accessing the source model still has a potential concern about leaking the source data, whic…
Domain AdaptationKnowledge DistillationSource-Free Domain AdaptationVocal Bursts Valence Prediction