Distilling Knowledge via Knowledge Review
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement. Code is available at https://github.com/Jia-Research-Lab/ReviewKD
Code (7)
Tasks
Instance SegmentationKnowledge Distillationobject-detectionObject DetectionSemantic SegmentationSimilar Papers 제목 키워드 기반
[Re] Distilling Knowledge via Knowledge Review
This effort aims to reproduce the results of experiments and analyze the robustness of the review framework for knowledge distillation introduced in the CVPR '21 paper 'Distilling Knowledge via Knowledge Review' by Chen …
A Comprehensive Survey on Knowledge Distillation of Diffusion Models
Diffusion Models (DMs), also referred to as score-based diffusion models, utilize neural networks to specify score functions. Unlike most other probabilistic models, DMs directly model the score functions, which makes th…
Knowledge DistillationSurveyA Survey on Extractive Knowledge Graph Summarization: Applications, Approaches, Evaluation, and Future Directions
With the continuous growth of large Knowledge Graphs (KGs), extractive KG summarization becomes a trending task. Aiming at distilling a compact subgraph with condensed information, it facilitates various downstream KG-ba…
Knowledge GraphsSurveyA Simple and Effective Self-Supervised Contrastive Learning Framework for Aspect Detection
Unsupervised aspect detection (UAD) aims at automatically extracting interpretable aspects and identifying aspect-specific segments (such as sentences) from online reviews. However, recent deep learning-based topic model…
Contrastive LearningTopic ModelsDistilling Word Embeddings: An Encoding Approach
Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restr…
Word Embeddings