Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language
This study introduces a prescriptive annotation benchmark grounded in humanities research to ensure consistent, unbiased labeling of offensive language, particularly for casual and non-mainstream language uses. We contribute two newly annotated datasets that achieve higher inter-annotator agreement between human and language model (LLM) annotations compared to original datasets based on descriptive instructions. Our experiments show that LLMs can serve as effective alternatives when professional annotators are unavailable. Moreover, smaller models fine-tuned on multi-source LLM-annotated data outperform models trained on larger, single-source human-annotated datasets. These findings highlight the value of structured guidelines in reducing subjective variability, maintaining performance with limited data, and embracing language diversity. Content Warning: This article only analyzes offensive language for academic purposes. Discretion is advised.
Code (0)
등록된 구현이 없습니다.
Tasks
DescriptiveDiversityLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Mitigating Bias in Facial Analysis Systems by Incorporating Label Diversity
Facial analysis models are increasingly applied in real-world applications that have significant impact on peoples' lives. However, as literature has shown, models that automatically classify facial attributes might exhi…
DiversityEnsemble LearningEvaluating Gender Bias in Speech Translation
The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of…
TranslationWisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds
Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often m…
DiversityDebiased Contrastive Representation Learning for Mitigating Dual Biases in Recommender Systems
In recommender systems, popularity and conformity biases undermine recommender effectiveness by disproportionately favouring popular items, leading to their over-representation in recommendation lists and causing an unba…
Contrastive LearningDiversityRecommendation SystemsRepresentation LearningUnderstanding and Mitigating Annotation Bias in Facial Expression Recognition
The performance of a computer vision model depends on the size and quality of its training data. Recent studies have unveiled previously-unknown composition biases in common image datasets which then lead to skewed model…
Facial Expression RecognitionFacial Expression Recognition (FER)Triplet