Logically Consistent Adversarial Attacks for Soft Theorem Provers
Recent efforts within the AI community have yielded impressive results towards "soft theorem proving" over natural language sentences using language models. We propose a novel, generative adversarial framework for probing and improving these models' reasoning capabilities. Adversarial attacks in this domain suffer from the logical inconsistency problem, whereby perturbations to the input may alter the label. Our Logically consistent AdVersarial Attacker, LAVA, addresses this by combining a structured generative process with a symbolic solver, guaranteeing logical consistency. Our framework successfully generates adversarial attacks and identifies global weaknesses common across multiple target models. Our analyses reveal naive heuristics and vulnerabilities in these models' reasoning capabilities, exposing an incomplete grasp of logical deduction under logic programs. Finally, in addition to effective probing of these models, we show that training on the generated samples improves the target model's performance.
Code (1)
Tasks
Automated Theorem ProvingSimilar Papers 제목 키워드 기반
Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
Backpropagation is the default algorithm for training deep neural networks due to its simplicity, efficiency and high convergence rate. However, its requirements make it impossible to be implemented in a human brain. In …
BenchmarkingExistence and Minimax Theorems for Adversarial Surrogate Risks in Binary Classification
Adversarial training is one of the most popular methods for training methods robust to adversarial attacks, however, it is not well-understood from a theoretical perspective. We prove and existence, regularity, and minim…
Adversarial RobustnessBinary ClassificationGenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models
We propose the first unified adversarial attack benchmark for Genomic Foundation Models (GFMs), named GenoArmory. Unlike existing GFM benchmarks, GenoArmory offers the first comprehensive evaluation framework to systemat…
Adversarial AttackAdversarial DefenseAdversarial RobustnessGenome Understanding+1BadSKP: Backdoor Attacks on Knowledge Graph-Enhanced LLMs with Soft Prompts
Recent knowledge graph (KG)-enhanced large language models (LLMs) move beyond purely textual knowledge augmentation by encoding retrieved subgraphs into continuous soft prompts via graph neural networks, introducing a gr…
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
We propose the first unified adversarial attack benchmark for Genomic Foundation Models (GFMs), named GERM. Unlike existing GFM benchmarks, GERM offers the first comprehensive evaluation framework to systematically asses…
Adversarial AttackAdversarial RobustnessComputational EfficiencyDNA analysis+5