A Comprehensive Review of Knowledge Distillation in Computer Vision
Deep learning techniques have been demonstrated to surpass preceding cutting-edge machine learning techniques in recent years, with computer vision being one of the most prominent examples. However, deep learning models suffer from significant drawbacks when deployed in resource-constrained environments due to their large model size and high complexity. Knowledge Distillation is one of the prominent solutions to overcome this challenge. This review paper examines the current state of research on knowledge distillation, a technique for compressing complex models into smaller and simpler ones. The paper provides an overview of the major principles and techniques associated with knowledge distillation and reviews the applications of knowledge distillation in the domain of computer vision. The review focuses on the benefits of knowledge distillation, as well as the problems that must be overcome to improve its effectiveness.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Comprehensive Survey on Knowledge Distillation
Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in…
Knowledge DistillationSurveyA Review on Discriminative Self-supervised Learning Methods in Computer Vision
Self-supervised learning (SSL) has rapidly emerged as a transformative approach in computer vision, enabling the extraction of rich feature representations from vast amounts of unlabeled data and reducing reliance on cos…
ClusteringKnowledge DistillationSelf-Supervised LearningTransfer LearningKnowledge Distillation: A Survey
In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale da…
Knowledge DistillationModel CompressionNeural Architecture SearchQuantization+1Recent Advances of Continual Learning in Computer Vision: An Overview
In contrast to batch learning where all training data is available at once, continual learning represents a family of methods that accumulate knowledge and learn continuously with data available in sequential order. Simi…
Continual LearningKnowledge DistillationData-Free Knowledge Transfer: A Survey
In the last decade, many deep learning models have been well trained and made a great success in various fields of machine intelligence, especially for computer vision and natural language processing. To better leverage …
Data-free Knowledge DistillationDomain AdaptationKnowledge DistillationModel Compression+3