Blind Knowledge Distillation for Robust Image Classification
Optimizing neural networks with noisy labels is a challenging task, especially if the label set contains real-world noise. Networks tend to generalize to reasonable patterns in the early training stages and overfit to specific details of noisy samples in the latter ones. We introduce Blind Knowledge Distillation - a novel teacher-student approach for learning with noisy labels by masking the ground truth related teacher output to filter out potentially corrupted knowledge and to estimate the tipping point from generalizing to overfitting. Based on this, we enable the estimation of noise in the training data with Otsus algorithm. With this estimation, we train the network with a modified weighted cross-entropy loss function. We show in our experiments that Blind Knowledge Distillation detects overfitting effectively during training and improves the detection of clean and noisy labels on the recently published CIFAR-N dataset. Code is available at GitHub.
Code (1)
Tasks
Classificationimage-classificationImage ClassificationKnowledge DistillationLearning with noisy labelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Knowledge Distillation based Degradation Estimation for Blind Super-Resolution
Blind image super-resolution (Blind-SR) aims to recover a high-resolution (HR) image from its corresponding low-resolution (LR) input image with unknown degradations. Most of the existing works design an explicit degrada…
Blind Super-ResolutionImage Super-ResolutionKnowledge DistillationSuper-ResolutionDistilling Spatially-Heterogeneous Distortion Perception for Blind Image Quality Assessment
In the Blind Image Quality Assessment (BIQA) field, accurately assessing the quality of authentically distorted images presents a substantial challenge due to the diverse distortion types in natural settings. Existin…
Blind Image Quality AssessmentImage Quality AssessmentKnowledge DistillationLocal DistortionLearning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment
An important scenario for image quality assessment (IQA) is to evaluate image restoration (IR) algorithms. The state-of-the-art approaches adopt a full-reference paradigm that compares restored images with their correspo…
Image Quality AssessmentImage RestorationKnowledge DistillationMonitored Distillation for Positive Congruent Depth Completion
We propose a method to infer a dense depth map from a single image, its calibration, and the associated sparse point cloud. In order to leverage existing models (teachers) that produce putative depth maps, we propose an …
Depth CompletionImage ReconstructionKnowledge DistillationModel SelectionDo Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation
Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output d…
Knowledge Distillation