Finding your ``Inner-Annotator'': An Experiment in Annotator Independence for Rating Discourse Coherence Quality in Essays
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets
Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them mass…
DiversityNatural Language UnderstandingAnnotator-Centric Active Learning for Subjective NLP Tasks
Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. However, for subjective NLP tasks, incorporating a wide range of perspectives in the…
Active LearningDiversityThe Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrum
In this work, we explore the capability of Large Language Models (LLMs) to annotate hate speech and abusiveness while considering predefined annotator personas within the strong-to-weak data perspectivism spectra. We eva…
Multi-annotator Deep Learning: A Probabilistic Framework for Classification
Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworker…
ClassificationDeep LearningIdentifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics
Machine learning is recently used to detect hate speech and other forms of abusive language in online platforms. However, a notable weakness of machine learning models is their vulnerability to bias, which can impair the…
Abusive LanguageBIG-bench Machine LearningFairness