Leveraging Annotator Disagreement for Text Classification
It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators. Doing so can remove valuable nuances and diverse perspectives inherent in the annotators' assessments. This paper proposes and compares three different strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning. All three approaches are evaluated on the tasks of hate speech and abusive conversation detection, which inherently entail a high degree of subjectivity. Moreover, to evaluate the effectiveness of embracing annotation disagreements for model training, we conduct an online survey that compares the performance of the multi-label model against a baseline model, which is trained with the majority label. The results show that in hate speech detection, the multi-label method outperforms the other two approaches, while in abusive conversation detection, instruction tuning achieves the best performance. The results of the survey also show that the outputs from the multi-label models are considered a better representation of the texts than the single-label model.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationHate Speech DetectionSurveytext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on the agreement of major annotators. However, annotators are individuals with different background…
A Decomposition-Based Approach for Evaluating Inter-Annotator Disagreement in Narrative Analysis
In this work, we explore sources of inter-annotator disagreement in narrative analysis, in light of the question of whether or not a narrative plot exists in the text. For this purpose, we present a method for a conceptu…
DH-FBK at SemEval-2022 Task 4: Leveraging Annotators’ Disagreement and Multiple Data Views for Patronizing Language Detection
The subtle and typically unconscious use of patronizing and condescending language (PCL) in large-audience media outlets undesirably feeds stereotypes and strengthens power-knowledge relationships, perpetuating discrimin…
Binary Condescension DetectionMulti-label Condescension DetectionLearning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts
Understanding moral values in social media text offers insight into moral judgement formation, and supervised NLP models trained on crowdsourced data have achieved strong classification performance. However, most approac…
Beyond Black \& White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in t…
Multi-Task Learning