Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments
We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.
Code (1)
Tasks
Fact CheckingMisinformationSimilar Papers 제목 키워드 기반
Early Detection of Misinformation for Infodemic Management: A Domain Adaptation Approach
An infodemic refers to an enormous amount of true information and misinformation disseminated during a disease outbreak. Detecting misinformation at the early stage of an infodemic is key to manage it and reduce its harm…
Domain AdaptationManagementMisinformationLarge Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability
Large language models (LLMs) have transformed misinformation from a primarily content-centric problem into a broader ecosystem-level security challenge. When misused, LLMs create risks beyond false content generation, en…
Contrastive Domain Adaptation for Early Misinformation Detection: A Case Study on COVID-19
Despite recent progress in improving the performance of misinformation detection systems, classifying misinformation in an unseen domain remains an elusive challenge. To address this issue, a common approach is to introd…
Domain AdaptationMisinformationXFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMs
The rapid spread of multimodal misinformation on social media calls for more effective and robust detection methods. Recent advances leveraging multimodal large language models (MLLMs) have shown the potential in address…
When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech
Hateful content online is often expressed using fact-like, not necessarily correct information, especially in coordinated online harassment campaigns and extremist propaganda. Failing to jointly address hate speech (HS) …