Robustness and Reliability When Training With Noisy Labels
Labelling of data for supervised learning can be costly and time-consuming and the risk of incorporating label noise in large data sets is imminent. When training a flexible discriminative model using a strictly proper loss, such noise will inevitably shift the solution towards the conditional distribution over noisy labels. Nevertheless, while deep neural networks have proven capable of fitting random labels, regularisation and the use of robust loss functions empirically mitigate the effects of label noise. However, such observations concern robustness in accuracy, which is insufficient if reliable uncertainty quantification is critical. We demonstrate this by analysing the properties of the conditional distribution over noisy labels for an input-dependent noise model. In addition, we evaluate the set of robust loss functions characterised by noise-insensitive, asymptotic risk minimisers. We find that strictly proper and robust loss functions both offer asymptotic robustness in accuracy, but neither guarantee that the final model is calibrated. Moreover, even with robust loss functions, overfitting is an issue in practice. With these results, we aim to explain observed robustness of common training practices, such as early stopping, to label noise. In addition, we aim to encourage the development of new noise-robust algorithms that not only preserve accuracy but that also ensure reliability.
Code (0)
등록된 구현이 없습니다.
Tasks
Uncertainty QuantificationSimilar Papers 제목 키워드 기반
Pathological Image Segmentation with Noisy Labels
Segmentation of pathological images is essential for accurate disease diagnosis. The quality of manual labels plays a critical role in segmentation accuracy; yet, in practice, the labels between pathologists could be inc…
Image SegmentationSegmentationSemantic SegmentationOn the Robustness of Monte Carlo Dropout Trained with Noisy Labels
The memorization effect of deep learning hinders its performance to effectively generalize on test set when learning with noisy labels. Prior study has discovered that epistemic uncertainty techniques are robust when tra…
Learning with noisy labelsMemorizationHow Does a Neural Network's Architecture Impact Its Robustness to Noisy Labels?
Noisy labels are inevitable in large real-world datasets. In this work, we explore an area understudied by previous works -- how the network's architecture impacts its robustness to noisy labels. We provide a formal fram…
Image ClassificationLearning with noisy labelsReliable Active Learning from Unreliable Labels via Neural Collapse Geometry
Active Learning (AL) promises to reduce annotation cost by prioritizing informative samples, yet its reliability is undermined when labels are noisy or when the data distribution shifts. In practice, annotators make mist…
Active LearningCalibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
The traditional process of creating labeled datasets is labor-intensive and expensive. Recent breakthroughs in open-source large language models (LLMs) have opened up a new avenue in generating labeled datasets automatic…
Benchmarking