BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification
In this work, we investigate the robustness of BERT using four word substitution-based attacks. We combine a human evaluation of individual word substitutions and a probabilistic analysis to show that between 96% and 99% of the analyzed attacks do not preserve semantics, indicating that their success is mainly based on feeding poor data to the model. To further confirm that, we introduce an efficient data augmentation procedure and show that many successful attacks can be prevented by including data similar to adversarial examples during training. Compared to traditional adversarial training, our data augmentation procedure requires 30x less computation time per epoch, while achieving better performance on two out of three datasets. We introduce an additional post-processing step that reduces the success rates of state-of-the-art attacks below 4%, 5%, and 8% on the three considered datasets. Finally, by looking at constraints for word substitutions that better preserve the semantics, we conclude that BERT is considerably more robust than previous research suggests.
Code (0)
등록된 구현이 없습니다.
Tasks
Data Augmentationtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Arabic Synonym BERT-based Adversarial Examples for Text Classification
Text classification systems have been proven vulnerable to adversarial text examples, modified versions of the original text examples that are often unnoticed by human eyes, yet can force text classification models to al…
Adversarial TextLanguage ModelingLanguage ModellingMasked Language Modeling+2BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification
Deep Neural Networks have taken Natural Language Processing by storm. While this led to incredible improvements across many tasks, it also initiated a new research field, questioning the robustness of these neural networ…
Data Augmentationtext-classificationText ClassificationDefense against Synonym Substitution-based Adversarial Attacks via Dirichlet Neighborhood Ensemble
Although deep neural networks have achieved prominent performance on many NLP tasks, they are vulnerable to adversarial examples. We propose Dirichlet Neighborhood Ensemble (DNE), a randomized method for training a robus…
SentenceFast Adversarial Training against Textual Adversarial Attacks
Many adversarial defense methods have been proposed to enhance the adversarial robustness of natural language processing models. However, most of them introduce additional pre-set linguistic knowledge and assume that the…
Adversarial DefenseAdversarial RobustnessBAE: BERT-based Adversarial Examples for Text Classification
Modern text classification models are susceptible to adversarial examples, perturbed versions of the original text indiscernible by humans which get misclassified by the model. Recent works in NLP use rule-based synonym …
Adversarial AttackAdversarial TextClassificationGeneral Classification+3