Detecting Label Errors by using Pre-Trained Language Models

2022-05-25 · Derek Chong, Jenny Hong, Christopher D. Manning

We show that large pre-trained language models are inherently highly capable of identifying label errors in natural language datasets: simply examining out-of-sample data points in descending order of fine-tuned task loss significantly outperforms more complex error-detection mechanisms proposed in previous work. To this end, we contribute a novel method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. We show that this noise has similar properties to real, hand-verified label errors, and is harder to detect than existing synthetic noise, creating challenges for model robustness. We argue that human-originated noise is a better standard for evaluation than synthetic noise. Finally, we use crowdsourced verification to evaluate the detection of real errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than existing models.

📄 PDF Abstract BibTeX arXiv:2205.12702

Code (1)

dcx/lnlfm 공식 구현

Detecting and Rectifying Noisy Labels: A Similarity-based Approach

2025-09-28 · Dang Huu-Tien, Minh-Phuong Nguyen, Naoya Inoue arxiv

Label noise in datasets could significantly damage the performance and robustness of deep neural networks (DNNs) trained on these datasets. As the size of modern DNNs grows, there is a growing demand for automated tools …

Detecting Label Errors in Token Classification Data

2022-10-08 · Wei-Chen Wang, Jonas Mueller

Mislabeled examples are a common issue in real-world data, particularly for tasks like token classification where many labels must be chosen on a fine-grained basis. Here we consider the task of finding sentences that co…

General Classification Token Classification

SELECT: Detecting Label Errors in Real-world Scene Text Data

2025-12-16 · Wenjun Liu, Qian Wu, Yifeng Hu, Yuke Li arxiv

We introduce SELECT (Scene tExt Label Errors deteCTion), a novel approach that leverages multi-modal training to detect label errors in real-world scene text datasets. Utilizing an image-text encoder and a character-leve…

Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

2024-10-24 · Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor 외

NLP benchmarks rely on standardized datasets for training and evaluating models and are crucial for advancing the field. Traditionally, expert annotations ensure high-quality labels; however, the cost of expert annotatio…

Detecting and refurbishing ground truth errors during training of deep learning-based echocardiography segmentation models

2026-04-14 · Iman Islam, Bram Ruijsink, Andrew J. Reader, Andrew P. King arxiv

Deep learning-based medical image segmentation typically relies on ground truth (GT) labels obtained through manual annotation, but these can be prone to random errors or systematic biases. This study examines the robust…

Medical Image Segmentation Label Error Detection