Iterative Refinement and Quality Checking of Annotation Guidelines --- How to Deal Effectively with Semantically Sloppy Named Entity Types, such as Pathological Phenomena
We here discuss a methodology for dealing with the annotation of semantically hard to delineate, i.e., sloppy, named entity types. To illustrate sloppiness of entities, we treat an example from the medical domain, namely pathological phenomena. Based on our experience with iterative guideline refinement we propose to carefully characterize the thematic scope of the annotation by positive and negative coding lists and allow for alternative, short vs. long mention span annotations. Short spans account for canonical entity mentions (e.g., standardized disease names), while long spans cover descriptive text snippets which contain entity-specific elaborations (e.g., anatomical locations, observational details, etc.). Using this stratified approach, evidence for increasing annotation performance is provided by kappa-based inter-annotator agreement measurements over several, iterative annotation rounds using continuously refined guidelines. The latter reflects the increasing understanding of the sloppy entity class both from the perspective of guideline writers and users (annotators). Given our data, we have gathered evidence that we can deal with sloppiness in a controlled manner and expect inter-annotator agreement values around 80{\%} for PathoJen, the pathological phenomena corpus currently under development in our lab.
Code (0)
등록된 구현이 없습니다.
Tasks
DescriptiveNamed Entity Recognition (NER)Similar Papers 제목 키워드 기반
Refining and Reusing Annotation Guidelines for LLM Annotation
While Large Language Models (LLMs) demonstrate remarkable performance on zero-shot annotation tasks, they often struggle with the specialized conventions of gold-standard benchmarks. We propose the systematic reuse and r…
Incremental Image Labeling via Iterative Refinement
Data quality is critical for multimedia tasks, while various types of systematic flaws are found in image benchmark datasets, as discussed in recent work. In particular, the existence of the semantic gap problem leads to…
Doomed to Re-Annotate, Forever: The ImageNet Story
Top-1 accuracy on ImageNet-1k remains the most commonly reported metric in visual recognition. Quality issues with the dataset have been repeatedly reported, yet the original 2012 noisy labels are still predominantly use…
Object LocalizationRepurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
This study investigates how existing annotation guidelines can be repurposed to instruct large language model (LLM) annotators for text annotation tasks. Traditional guidelines are written for human annotators who intern…
PerCQA: Persian Community Question Answering Dataset
Community Question Answering (CQA) forums provide answers for many real-life questions. Thanks to the large size, these forums are very popular among machine learning researchers. Automatic answer selection, answer ranki…
Answer SelectionCommunity Question AnsweringFact CheckingQuestion Answering+1