Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.
Code (0)
등록된 구현이 없습니다.
Tasks
Value predictionSimilar Papers 제목 키워드 기반
Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities
The prevalence and impact of toxic discussions online have made content moderation crucial.Automated systems can play a vital role in identifying toxicity, and reducing the reliance on human moderation.Nevertheless, iden…
Language ModelingLanguage ModellingLarge Language ModelLearning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts
Understanding moral values in social media text offers insight into moral judgement formation, and supervised NLP models trained on crowdsourced data have achieved strong classification performance. However, most approac…
You Are What You Annotate: Towards Better Models through Annotator Representations
Annotator disagreement is ubiquitous in natural language processing (NLP) tasks. There are multiple reasons for such disagreements, including the subjectivity of the task, difficult cases, unclear guidelines, and so on. …
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been see…
PositionD3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examined this issue are often situated in the …
4k