Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective.
Code (1)
Tasks
Key Information Extractionnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERSimilar Papers 제목 키워드 기반
TASTEset -- Recipe Dataset and Food Entities Recognition Benchmark
Food Computing is currently a fast-growing field of research. Natural language processing (NLP) is also increasingly essential in this field, especially for recognising food entities. However, there are still only a few …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERGLIMG: Global and Local Item Graphs for Top-N Recommender Systems
Graph-based recommendation models work well for top-N recommender systems due to their capability to capture the potential relationships between entities. However, most of the existing methods only construct a single glo…
Recommendation SystemsNot-quite-human tastes: the stylized omnivorousness of LLM survey surrogates
Large-language models have proven to be remarkable if inconsistent parrots of public attitudes and opinions. The extent to which LLMs are able to produce reasonable approximations of cultural taste remains an open empiri…
Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs
Knowledge Graphs (KGs) play a crucial role in enhancing e-commerce system performance by providing structured information about entities and their relationships, such as complementary or substitutable relations between p…
Knowledge Graph CompletionKnowledge GraphsPrompt EngineeringRecommendation Systems+1Mixture-of-tastes Models for Representing Users with Diverse Interests
Most existing recommendation approaches implicitly treat user tastes as unimodal, resulting in an average-of-tastes representations when multiple distinct interests are present. We show that appropriately modelling the m…