Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Machine learning algorithms are often vulnerable to adversarial examples that have imperceptible alterations from the original counterparts but can fool the state-of-the-art models. It is helpful to evaluate or even improve the robustness of these models by exposing the maliciously crafted adversarial examples. In this paper, we present TextFooler, a simple but strong baseline to generate natural adversarial text. By applying it to two fundamental natural language tasks, text classification and textual entailment, we successfully attacked three target models, including the powerful pre-trained BERT, and the widely used convolutional and recurrent neural networks. We demonstrate the advantages of this framework in three ways: (1) effective---it outperforms state-of-the-art attacks in terms of success rate and perturbation rate, (2) utility-preserving---it preserves semantic content and grammaticality, and remains correctly classified by humans, and (3) efficient---it generates adversarial text with computational complexity linear to the text length. *The code, pre-trained target models, and test examples are available at https://github.com/jind11/TextFooler.
Code (7)
Tasks
Adversarial TextGeneral ClassificationNatural Language InferenceText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How Fast can BERT Learn Simple Natural Language Inference?
This paper empirically studies whether BERT can really learn to conduct natural language inference (NLI) without utilizing hidden dataset bias; and how efficiently it can learn if it could. This is done via creating a si…
Natural Language InferenceDoes BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models obtain impressive results on a wide variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb num…
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models demonstrate impressive performance on a variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb…
Exponentially Faster Language Modelling
Language models only really need to use an exponential fraction of their neurons for individual inferences. As proof, we present UltraFastBERT, a BERT variant that uses 0.3% of its neurons during inference while performi…
BenchmarkingCPULanguage ModellingMultilingual Question Answering from Formatted Text applied to Conversational Agents
Recent advances with language models (e.g. BERT, XLNet, ...), have allowed surpassing human performance on complex NLP tasks such as Reading Comprehension. However, labeled datasets for training are available mostly in E…
Extractive Question-AnsweringMachine TranslationQuestion AnsweringReading Comprehension