Structured Knowledge Distillation for Semantic Segmentation
In this paper, we investigate the issue of knowledge distillation for training compact semantic segmentation networks by making use of cumbersome networks. We start from the straightforward scheme, pixel-wise distillation, which applies the distillation scheme originally introduced for image classification and performs knowledge distillation for each pixel separately. We further propose to distill the structured knowledge from cumbersome networks into compact networks, which is motivated by the fact that semantic segmentation is a structured prediction problem. We study two such structured distillation schemes: (i) pair-wise distillation that distills the pairwise similarities, and (ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by extensive experiments on three scene parsing datasets: Cityscapes, Camvid and ADE20K.
Code (1)
Tasks
General Classificationimage-classificationImage ClassificationKnowledge DistillationScene ParsingSegmentationSemantic SegmentationStructured PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cross-Image Relational Knowledge Distillation for Semantic Segmentation
Current Knowledge Distillation (KD) methods for semantic segmentation often guide the student to mimic the teacher's structured information generated from individual data samples. However, they ignore the global semantic…
Knowledge DistillationSegmentationSemantic SegmentationStructured Knowledge Distillation for Dense Prediction
In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tas…
Depth EstimationGeneral Classificationimage-classificationImage Classification+7FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentation
In this work, we explore data augmentations for knowledge distillation on semantic segmentation. To avoid over-fitting to the noise in the teacher network, a large number of training examples is essential for knowledge d…
Knowledge DistillationSegmentationSemantic SegmentationAugmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less atten…
Contrastive LearningData Augmentationimage-classificationImage Classification+5Distilling Inter-Class Distance for Semantic Segmentation
Knowledge distillation is widely adopted in semantic segmentation to reduce the computation cost.The previous knowledge distillation methods for semantic segmentation focus on pixel-wise feature alignment and intra-class…
Knowledge DistillationPositionSegmentationSemantic Segmentation