Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios
Knowledge Distillation (KD) transfers knowledge from a large pre-trained teacher network to a compact and efficient student network, making it suitable for deployment on resource-limited media terminals. However, traditional KD methods require balanced data to ensure robust training, which is often unavailable in practical applications. In such scenarios, a few head categories occupy a substantial proportion of examples. This imbalance biases the trained teacher network towards the head categories, resulting in severe performance degradation on the less represented tail categories for both the teacher and student networks. In this paper, we propose a novel framework called Knowledge Rectification Distillation (KRDistill) to address the imbalanced knowledge inherited in the teacher network through the incorporation of the balanced category priors. Furthermore, we rectify the biased predictions produced by the teacher network, particularly focusing on the tail categories. Consequently, the teacher network can provide balanced and accurate knowledge to train a reliable student network. Intensive experiments conducted on various long-tailed datasets demonstrate that our KRDistill can effectively train reliable student networks in realistic scenarios of data imbalance.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationTransfer LearningSimilar Papers 제목 키워드 기반
Rectifying the Data Bias in Knowledge Distillation
Knowledge distillation is a representative technique for model compression and acceleration, which is important for deploying neural networks on resource limited devices. The knowledge transferred from teacher to stud…
Face RecognitionFace VerificationKnowledge DistillationModel Compression+1Domain-Aware Contrastive Knowledge Transfer for Multi-domain Imbalanced Data
In many real-world machine learning applications, samples belong to a set of domains e.g., for product reviews each review belongs to a product category. In this paper, we study multi-domain imbalanced learning (MIL), th…
Transfer LearningDecay Pruning Method: Smooth Pruning With a Self-Rectifying Procedure
Current structured pruning methods often result in considerable accuracy drops due to abrupt network changes and loss of information from pruned structures. To address these issues, we introduce the Decay Pruning Method …
Rectify the Regression Bias in Long-Tailed Object Detection
Long-tailed object detection faces great challenges because of its extremely imbalanced class distribution. Recent methods mainly focus on the classification bias and its loss function design, while ignoring the subtle i…
Long-tailed Object DetectionObjectobject-detectionObject Detection+1Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation
Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets a…