Does label smoothing mitigate label noise?
Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem --- being equivalent to injecting symmetric noise to the labels --- we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning with noisy labelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Exploration into why Output Regularization Mitigates Label Noise
Label noise presents a real challenge for supervised learning algorithms. Consequently, mitigating label noise has attracted immense research in recent years. Noise robust losses is one of the more promising approaches f…
Graph Neural Diffusion Networks for Semi-supervised Learning
Graph Convolutional Networks (GCN) is a pioneering model for graph-based semi-supervised learning. However, GCN does not perform well on sparsely-labeled graphs. Its two-layer version cannot effectively propagate the lab…
Graph Neural NetworkWhen Does Label Smoothing Help?
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Sm…
image-classificationImage ClassificationKnowledge Distillationspeech-recognition+2Fighting noise and imbalance in Action Unit detection problems
Action Unit (AU) detection aims at automatically caracterizing facial expressions with the muscular activations they involve. Its main interest is to provide a low-level face representation that can be used to assist hig…
Action Unit DetectionImproving Time Series Classification with Representation Soft Label Smoothing
Previous research has indicated that deep neural network based models for time series classification (TSC) tasks are prone to overfitting. This issue can be mitigated by employing strategies that prevent the model from b…
Time SeriesTime Series Classification