Better Generalization with On-the-fly Dataset Denoising
Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are to hard avoid in extremely large datasets. We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates, which we find to be able to separate clean and mislabeled examples with remarkable success using loss statistics. We leverage this to identify and on-the-fly discard mislabeled examples using a threshold on their losses. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead. Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingMemorizationSimilar Papers 제목 키워드 기반
Denoising after Entropy-based Debiasing A Robust Training Method for Dataset Bias with Noisy Labels
Improperly constructed datasets can result in inaccurate inferences. For instance, models trained on biased datasets perform poorly in terms of generalization (i.e., dataset bias). Recent debiasing techniques have succes…
DenoisingWINNet: Wavelet-inspired Invertible Network for Image Denoising
Image denoising aims to restore a clean image from an observed noisy image. The model-based image denoising approaches can achieve good generalization ability over different noise levels and are with high interpretabilit…
DeblurringDenoisingImage DeblurringImage Denoising+1Masked Image Training for Generalizable Deep Image Denoising
When capturing and storing images, devices inevitably introduce noise. Reducing this noise is a critical task called image denoising. Deep learning has become the de facto method for image denoising, especially with the …
Deep LearningDenoisingImage DenoisingLearning to Drop: Robust Graph Neural Network via Topological Denoising
Graph Neural Networks (GNNs) have shown to be powerful tools for graph analytics. The key idea is to recursively propagate and aggregate information along edges of the given graph. Despite their success, however, the exi…
DenoisingGraph Neural NetworkLink PredictionNode ClassificationCoreDiff: Contextual Error-Modulated Generalized Diffusion Model for Low-Dose CT Denoising and Generalization
Low-dose computed tomography (CT) images suffer from noise and artifacts due to photon starvation and electronic noise. Recently, some works have attempted to use diffusion models to address the over-smoothness and train…
Computed Tomography (CT)DenoisingOne-Shot Learning