Towards a Unified View of Affinity-Based Knowledge Distillation
Knowledge transfer between artificial neural networks has become an important topic in deep learning. Among the open questions are what kind of knowledge needs to be preserved for the transfer, and how it can be effectively achieved. Several recent work have shown good performance of distillation methods using relation-based knowledge. These algorithms are extremely attractive in that they are based on simple inter-sample similarities. Nevertheless, a proper metric of affinity and use of it in this context is far from well understood. In this paper, by explicitly modularising knowledge distillation into a framework of three components, i.e. affinity, normalisation, and loss, we give a unified treatment of these algorithms as well as study a number of unexplored combinations of the modules. With this framework we perform extensive evaluations of numerous distillation objectives for image classification, and obtain a few useful insights for effective design choices while demonstrating how relation-based knowledge distillation could achieve comparable performance to the state of the art in spite of the simplicity.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationKnowledge DistillationRelationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
VRM: Knowledge Distillation via Virtual Relation Matching
Knowledge distillation (KD) aims to transfer the knowledge of a more capable yet cumbersome teacher model to a lightweight student model. In recent years, relation-based KD methods have fallen behind, as their instance-m…
Knowledge DistillationRelationInvestigating Knowledge Distillation Through Neural Networks for Protein Binding Affinity Prediction
The trade-off between predictive accuracy and data availability makes it difficult to predict protein--protein binding affinity accurately. The lack of experimentally resolved protein structures limits the performance of…
Knowledge DistillationGraph Relation Distillation for Efficient Biomedical Instance Segmentation
Instance-aware embeddings predicted by deep neural networks have revolutionized biomedical instance segmentation, but its resource requirements are substantial. Knowledge distillation offers a solution by transferring di…
Instance SegmentationKnowledge DistillationRelationSemantic SegmentationHigh-order Correlation Preserved Incomplete Multi-view Subspace Clustering
Incomplete multi-view clustering aims to exploit theinformation of multiple incomplete views to partition data into their clusters. Existing methods only utilize the pair-wise sample correlation and pair-wise view correl…
ClusteringIncomplete multi-view clusteringMulti-view Subspace ClusteringVocal Bursts Intensity PredictionMulti-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation
3D point cloud semantic segmentation is one of the fundamental tasks for environmental understanding. Although significant progress has been made in recent years, the performance of classes with few examples or few point…
Knowledge DistillationSemantic Segmentation