Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised training is performed using the full (noisy) training set. In this paper, we identify a "warm-up obstacle": the inability of standard warm-up stages to train high quality feature extractors and avert memorization of noisy labels. We propose "Contrast to Divide" (C2D), a simple framework that solves this problem by pre-training the feature extractor in a self-supervised fashion. Using self-supervised pre-training boosts the performance of existing LNL approaches by drastically reducing the warm-up stage's susceptibility to noise level, shortening its duration, and improving extracted feature quality. C2D works out of the box with existing methods and demonstrates markedly improved performance, especially in the high noise regime, where we get a boost of more than 27% for CIFAR-100 with 90% noise over the previous state of the art. In real-life noise settings, C2D trained on mini-WebVision outperforms previous works both in WebVision and ImageNet validation sets by 3% top-1 accuracy. We perform an in-depth analysis of the framework, including investigating the performance of different pre-training approaches and estimating the effective upper bound of the LNL performance with semi-supervised learning. Code for reproducing our experiments is available at https://github.com/ContrastToDivide/C2D
Code (1)
Tasks
Image ClassificationLearning with noisy labelsMemorizationSimilar Papers 제목 키워드 기반
Manifold DivideMix: A Semi-Supervised Contrastive Learning Framework for Severe Label Noise
Deep neural networks have proven to be highly effective when large amounts of data with clean labels are available. However, their performance degrades when training data contains noisy labels, leading to poor generaliza…
Contrastive LearningImproving Medical Image Classification in Noisy Labels Using Only Self-supervised Pretraining
Noisy labels hurt deep learning-based supervised image classification performance as the models may overfit the noise and learn corrupted feature extractors. For natural image classification training with noisy labeled d…
Classificationimage-classificationImage ClassificationLearning with noisy labels+2Divide and Contrast: Self-supervised Learning from Uncurated Data
Self-supervised learning holds promise in leveraging large amounts of unlabeled data, however much of its progress has thus far been limited to highly curated pre-training data such as ImageNet. We explore the effects of…
ClusteringContrastive LearningSelf-Supervised Image ClassificationSelf-Supervised LearningCoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
Labels are costly and sometimes unreliable. Noisy label learning, semi-supervised learning, and contrastive learning are three different strategies for designing learning processes requiring less annotation cost. Semi-su…
Contrastive LearningImage ClassificationLearning with noisy labelsNoise2Contrast: Multi-Contrast Fusion Enables Self-Supervised Tomographic Image Denoising
Self-supervised image denoising techniques emerged as convenient methods that allow training denoising models without requiring ground-truth noise-free data. Existing methods usually optimize loss metrics that are calcul…
DenoisingImage DenoisingSSIM