MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification
We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies.
Code (1)
Tasks
Zero-Shot LearningSimilar Papers 제목 키워드 기반
Are LLMs Good Zero-Shot Fallacy Classifiers?
Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers ar…
MisinformationEmotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection
Logical fallacies are common in public communication and can mislead audiences; fallacious arguments may still appear convincing despite lacking soundness, because convincingness is inherently subjective. We present the …
Logical FallaciesFine-grained Fallacy Detection with Human Label Variation
We introduce Faina, the first dataset for fallacy detection that embraces multiple plausible answers and natural disagreement. Faina includes over 11K span-level annotations with overlaps across 20 fallacy types on socia…
Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies
Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored. Prior work has primarily examined whether LLMs …
Logical FallaciesSocrates or Smartypants: Testing Logic Reasoning Capabilities of Large Language Models with Logic Programming-based Test Oracles
Large Language Models (LLMs) have achieved significant progress in language understanding and reasoning. Evaluating and analyzing their logical reasoning abilities has therefore become essential. However, existing datase…
Logical FallaciesLogical Reasoning