Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection
The proliferation of radical content on online platforms poses significant risks, including inciting violence and spreading extremist ideologies. Despite ongoing research, existing datasets and models often fail to address the complexities of multilingual and diverse data. To bridge this gap, we introduce a publicly available multilingual dataset annotated with radicalization levels, calls for action, and named entities in English, French, and Arabic. This dataset is pseudonymized to protect individual privacy while preserving contextual information. Beyond presenting our freely available dataset, we analyze the annotation process, highlighting biases and disagreements among annotators and their implications for model performance. Additionally, we use synthetic data to investigate the influence of socio-demographic traits on annotation patterns and model predictions. Our work offers a comprehensive examination of the challenges and opportunities in building robust datasets for radical content detection, emphasizing the importance of fairness and transparency in model development.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessSimilar Papers 제목 키워드 기반
Dataset creation for supervised deep learning-based analysis of microscopic images -- review of important considerations and recommendations
Supervised deep learning (DL) receives great interest for automated analysis of microscopic images with an increasing body of literature supporting its potential. The development and validation of those DL models relies …
WildAvatar: Web-scale In-the-wild Video Dataset for 3D Avatar Creation
Existing human datasets for avatar creation are typically limited to laboratory environments, wherein high-quality annotations (e.g., SMPL estimation from 3D scans or multi-view images) can be ideally provided. However, …
Algorithmic Ghost in the Research Shell: Large Language Models and Academic Knowledge Creation in Management Research
The paper looks at the role of large language models in academic knowledge creation based on a scoping review (2018 to January 2023) of how researchers have previously used the language model GPT to assist in the perform…
ArticlesLanguage ModelingLanguage ModellingManagementBuilding an annotated dataset of app store reviews with Appraisal features in English and Spanish
This paper describes the creation and annotation of a dataset consisting of 250 English and Spanish app store reviews from Google{'}s Play Store with Appraisal features. This is one of the most influential linguistic fra…
DescriptiveGeneral ClassificationBeyond Explanation: A Case for Exploratory Text Visualizations of Non-Aggregated, Annotated Datasets
This paper presents an overview of text visualization techniques relevant for data perspectivism, aiming to facilitate analysis of annotated datasets for the datasets’ creators and stakeholders. Data perspectivism advoca…
Bias DetectionHate Speech Detectionvalid