Evaluating adversarial attacks against multiple fact verification systems
Automated fact verification has been progressing owing to advancements in modeling and availability of large datasets. Due to the nature of the task, it is critical to understand the vulnerabilities of these systems against adversarial instances designed to make them predict incorrectly. We introduce two novel scoring metrics, attack potency and system resilience which take into account the correctness of the adversarial instances, an aspect often ignored in adversarial evaluations. We consider six fact verification systems from the recent Fact Extraction and VERification (FEVER) challenge: the four best-scoring ones and two baselines. We evaluate adversarial instances generated by a recently proposed state-of-the-art method, a paraphrasing method, and rule-based attacks devised for fact verification. We find that our rule-based attacks have higher potency, and that while the rankings among the top systems changed, they exhibited higher resilience than the baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Fact VerificationSimilar Papers 제목 키워드 기반
On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs
This research investigates the effectiveness of established vulnerability metrics, such as the Common Vulnerability Scoring System (CVSS), in evaluating attacks against Large Language Models (LLMs), with a focus on Adver…
On Evaluating the Adversarial Robustness of Semantic Segmentation Models
Achieving robustness against adversarial input perturbation is an important and intriguing problem in machine learning. In the area of semantic image segmentation, a number of adversarial training approaches have been pr…
Adversarial Robustnessimage-classificationImage ClassificationImage Segmentation+1Adversarial Attacks Against Automated Fact-Checking: A Survey
In an era where misinformation spreads freely, fact-checking (FC) plays a crucial role in verifying claims and promoting reliable information. While automated fact-checking (AFC) has advanced significantly, existing syst…
Perturbation Type Categorization for Multiple $\ell_p$ Bounded Adversarial Robustness
Despite the recent advances in $\textit{adversarial training}$ based defenses, deep neural networks are still vulnerable to adversarial attacks outside the perturbation type they are trained to be robust against. Recent …
Adversarial RobustnessVocal Bursts Type PredictionRobCaps: Evaluating the Robustness of Capsule Networks against Affine Transformations and Adversarial Attacks
Capsule Networks (CapsNets) are able to hierarchically preserve the pose relationships between multiple objects for image classification tasks. Other than achieving high accuracy, another relevant factor in deploying Cap…
image-classificationImage Classification