Detecting Label Errors in Token Classification Data
Mislabeled examples are a common issue in real-world data, particularly for tasks like token classification where many labels must be chosen on a fine-grained basis. Here we consider the task of finding sentences that contain label errors in token classification datasets. We study 11 different straightforward methods that score tokens/sentences based on the predicted class probabilities output by a (any) token classification model (trained via any procedure). In precision-recall evaluations based on real-world label errors in entity recognition data from CoNLL-2003, we identify a simple and effective method that consistently detects those sentences containing label errors when applied with different token classification models.
Code (2)
Tasks
General ClassificationToken ClassificationSimilar Papers 제목 키워드 기반
SELECT: Detecting Label Errors in Real-world Scene Text Data
We introduce SELECT (Scene tExt Label Errors deteCTion), a novel approach that leverages multi-modal training to detect label errors in real-world scene text datasets. Utilizing an image-text encoder and a character-leve…
SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization
NLP datasets may still contain annotation errors, even when they are manually annotated. Researchers have attempted to develop methods to automatically reduce the adverse effect of errors in datasets. However, existing m…
Document Classificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+4Entity at SemEval-2021 Task 5: Weakly Supervised Token Labelling for Toxic Spans Detection
Detection of toxic spans - detecting toxicity of contents in the granularity of tokens - is crucial for effective moderation of online discussions. The baseline approach for this problem using the transformer model is to…
ClassificationLanguage ModelingLanguage ModellingSentence+4Identifying Label Errors in Object Detection Datasets by Loss Inspection
Labeling datasets for supervised object detection is a dull and time-consuming task. Errors can be easily introduced during annotation and overlooked during review, yielding inaccurate benchmarks and performance degradat…
Label Error DetectionObjectobject-detectionObject DetectionRevising RVL-CDIP: Quantifying Errors and Test-Train Overlap
RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model perfor…
Document Classification