Noise against noise: stochastic label noise helps combat inherent label noise
The noise in stochastic gradient descent (SGD) provides a crucial implicit regularization effect, previously studied in optimization by analyzing the dynamics of parameter updates. In this paper, we are interested in learning with noisy labels, where we have a collection of samples with potential mislabeling. We show that a previously rarely discussed SGD noise, induced by stochastic label noise (SLN), mitigates the effects of inherent label noise. In contrast, the common SGD noise directly applied to model parameters does not. We formalize the differences and connections of SGD noise variants, showing that SLN induces SGD noise dependent on the sharpness of output landscape and the confidence of output probability, which may help escape from sharp minima and prevent overconfidence. SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample-selection and label-correction. Specifically, we present an enhanced algorithm by applying SLN to label-correction.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning with noisy labelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability
Random label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to…
Implicit Regularization Effects of Unbiased Random Label Noises with SGD
Random label noises (or observational noises) widely exist in practical machinelearning settings. we analyze the learning dynamics of stochastic gradient descent(SGD) over the quadratic loss with unbiased label noises, a…
Simple and Effective Stochastic Neural Networks
Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized lear…
Adversarial AttackAdversarial DefenseTrustNet: Learning from Trusted Data Against (A)symmetric Label Noise
Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datasets. Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datase…
Handling Realistic Label Noise in BERT Text Classification
Labels noise refers to errors in training labels caused by cheap data annotation methods, such as web scraping or crowd-sourcing, which can be detrimental to the performance of supervised classifiers. Several methods hav…
Classificationtext-classificationText Classification