Structured Knowledge Distillation for Dense Prediction

2019-03-11 · CVPR 2019 6 · Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen

In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at: https://git.io/StructKD

📄 PDF Abstract BibTeX arXiv:1903.04197

Code (1)

irfanICMLL/structure_knowledge_distillation 공식 구현 pytorch

Tasks

Depth Estimation General Classification image-classification Image Classification Knowledge Distillation object-detection Object Detection Prediction Scene Parsing Semantic Segmentation Structured Prediction

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Structured Knowledge Distillation for Semantic Segmentation

2019-06-01 · CVPR 2019 6 · Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin 외

In this paper, we investigate the issue of knowledge distillation for training compact semantic segmentation networks by making use of cumbersome networks. We start from the straightforward scheme, pixel-wise distillatio…

General Classification image-classification Image Classification Knowledge Distillation+4

Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation

2023-12-07 · Jiawei Fan, Chao Li, Xiaolong Liu, Meina Song 외

In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less atten…

Contrastive Learning Data Augmentation image-classification Image Classification+5

Augmentation-free Dense Contrastive Distillation for Efficient Semantic Segmentation

2023-09-21 · NeurIPS 2023 11

Multi-Task Learning with Knowledge Distillation for Dense Prediction

2023-01-01 · ICCV 2023 1 · Yangyang Xu, Yibo Yang, Lefei Zhang

While multi-task learning (MTL) has become an attractive topic, its training usually poses more difficulties than the single-task case. How to successfully apply knowledge distillation into MTL to improve training ef…

Boundary Detection Depth Estimation Knowledge Distillation Multi-Task Learning+2

Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions

2022-05-30 · CVPR 2023 1 · Shuxuan Guo, Yinlin Hu, Jose M. Alvarez, Mathieu Salzmann

Knowledge distillation facilitates the training of a compact student network by using a deep teacher one. While this has achieved great success in many tasks, it remains completely unstudied for image-based 6D object pos…

6D Pose Estimation 6D Pose Estimation using RGB Knowledge Distillation Pose Estimation+1