Design-Based Supervised Learning with Noisy Human Labels
Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Self-semi-supervised Learning to Learn from NoisyLabeled Data
The remarkable success of today's deep neural networks highly depends on a massive number of correctly labeled data. However, it is rather costly to obtain high-quality human-labeled data, leading to the active research …
Semi-Supervised Cascaded Clustering for Classification of Noisy Label Data
The performance of supervised classification techniques often deteriorates when the data has noisy labels. Even the semi-supervised classification approaches have largely focused only on the problem of handling missing l…
ClusteringMissing LabelsImproving Medical Image Classification in Noisy Labels Using Only Self-supervised Pretraining
Noisy labels hurt deep learning-based supervised image classification performance as the models may overfit the noise and learn corrupted feature extractors. For natural image classification training with noisy labeled d…
Classificationimage-classificationImage ClassificationLearning with noisy labels+2Combining Self-Supervised and Supervised Learning with Noisy Labels
Since convolutional neural networks (CNNs) can easily overfit noisy labels, which are ubiquitous in visual classification tasks, it has been a great challenge to train CNNs against them robustly. Various methods have bee…
Learning with noisy labelsRepresentation LearningSelf-Supervised LearningStudent as an Inherent Denoiser of Noisy Teacher
Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by t…
Knowledge DistillationLanguage ModelingLanguage ModellingLarge Language Model+1