Turning silver into gold: error-focused corpus reannotation with active learning
While high quality gold standard annotated corpora are crucial for most tasks in natural language processing, many annotated corpora published in recent years, created by annotators or tools, contains noisy annotations. These corpora can be viewed as more silver than gold standards, even if they are used in evaluation campaigns or to compare systems{'} performances. As upgrading a silver corpus to gold level is still a challenge, we explore the application of active learning techniques to detect errors using four datasets designed for document classification and part-of-speech tagging. Our results show that the proposed method for the seeding step improves the chance of finding incorrect annotations by a factor of 2.73 when compared to random selection, a 14.71{\%} increase from the baseline methods. Our query method provides an increase in the error detection precision on average by a factor of 1.78 against random selection, an increase of 61.82{\%} compared to other query approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningDocument ClassificationPart-Of-Speech TaggingSimilar Papers 제목 키워드 기반
Turning Silver into Gold: Domain Adaptation with Noisy Labels for Wearable Cardio-Respiratory Fitness Prediction
Deep learning models have shown great promise in various healthcare applications. However, most models are developed and validated on small-scale datasets, as collecting high-quality (gold-standard) labels for health app…
Domain AdaptationUnsupervised Domain AdaptationSmelting Gold and Silver for Improved Multilingual AMR-to-Text Generation
Recent work on multilingual AMR-to-text generation has exclusively focused on data augmentation strategies that utilize silver AMR. However, this assumes a high quality of generated AMRs, potentially limiting the transfe…
AMR-to-Text GenerationData AugmentationText GenerationSilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment
Word alignments are essential for a variety of NLP tasks. Therefore, choosing the best approaches for their creation is crucial. However, the scarce availability of gold evaluation data makes the choice difficult. We pro…
Machine TranslationTranslationvalidWord AlignmentCentroids: Gold standards with distributional variation
Motivation: Gold Standards for named entities are, ironically, not standard themselves. Some specify the one perfect annotation. Others specify perfectly good alternatives. The concept of Silver standard is relativel…
Named Entity Recognition (NER)When silver glitters more than gold: Bootstrapping an Italian part-of-speech tagger for Twitter
We bootstrap a state-of-the-art part-of-speech tagger to tag Italian Twitter data, in the context of the Evalita 2016 PoSTWITA shared task. We show that training the tagger on native Twitter data enriched with little amo…
TAG