Adversarial Examples for Evaluating Math Word Problem Solvers
Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40 percentage points on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.
Code (1)
Tasks
Adversarial RobustnessMathMath Word Problem SolvingSentenceSimilar Papers 제목 키워드 기반
Adversarial Math Word Problem Generation
Large language models (LLMs) have significantly transformed the educational landscape. As current plagiarism detection tools struggle to keep pace with LLMs' rapid advancements, the educational community faces the challe…
MathEvaluating the Validity of Word-level Adversarial Attacks with Large Language Models
Deep neural networks exhibit vulnerability to word-level adversarial attacks in natural language processing. Most of these attack methods adopt synonymous substitutions to perturb original samples for crafting adversaria…
Adversarial AttackLanguage ModelingLanguage ModellingLarge Language Model+1Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input …
image-classificationImage ClassificationMachine TranslationText Summarization+1On Adversarial Examples for Character-Level Neural Machine Translation
Evaluating on adversarial examples has become a standard procedure to measure robustness of deep learning models. Due to the difficulty of creating white-box adversarial examples for discrete text input, most analyses of…
Machine TranslationNMTTranslationTraining Meta-Surrogate Model for Transferable Adversarial Attack
We consider adversarial attacks to a black-box model when no queries are allowed. In this setting, many methods directly attack surrogate models and transfer the obtained adversarial examples to fool the target model. Pl…
Adversarial Attackmodel