CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English
Modern named entity recognition systems have steadily improved performance in the age of larger and more powerful neural models. However, over the past several years, the state-of-the-art has seemingly hit another plateau on the benchmark CoNLL-03 English dataset. In this paper, we perform a deep dive into the test outputs of the highest-performing NER models, conducting a fine-grained evaluation of their performance by introducing new document-level annotations on the test set. We go beyond F1 scores by categorizing errors in order to interpret the true state of the art for NER and guide future work. We review previous attempts at correcting the various flaws of the test set and introduce CoNLL#, a new corrected version of the test set that addresses its systematic and most prevalent errors, allowing for low-noise, interpretable error analysis.
Code (0)
등록된 구현이 없습니다.
Tasks
named-entity-recognitionNamed Entity RecognitionNERMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset
The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistenc…
Entity Linkingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Until now, error type performance for Grammatical Error Correction (GEC) systems could only be measured in terms of recall because system output is not annotated. To overcome this problem, we introduce ERRANT, a grammati…
Annotated Code SearchGrammatical Error CorrectionNews AnnotationTable annotation+1Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
Neural sequence-to-sequence (seq2seq) approaches have proven to be successful in grammatical error correction (GEC). Based on the seq2seq framework, we propose a novel fluency boost learning and inference mechanism. Flue…
Grammatical Error CorrectionSentenceIdentifying Incorrect Labels in the CoNLL-2003 Corpus
The CoNLL-2003 corpus for English-language named entity recognition (NER) is one of the most influential corpora for NER model research. A large number of publications, including many landmark works, have used this corpu…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERA Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction
Existing approaches for grammatical error correction (GEC) largely rely on supervised learning with manually created GEC datasets. However, there has been little focus on verifying and ensuring the quality of the dataset…
DenoisingGrammatical Error Correction