Towards Robustness of Text-to-SQL Models against Synonym Substitution
Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.
Code (1)
Tasks
Text to SQLText-To-SQLSimilar Papers 제목 키워드 기반
Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks
The language models, especially the basic text classification models, have been shown to be susceptible to textual adversarial attacks such as synonym substitution and word insertion attacks. To defend against such attac…
text-classificationText ClassificationAdversarial Training with Fast Gradient Projection Method against Synonym Substitution based Text Attacks
Adversarial training is the most empirically successful approach in improving the robustness of deep neural networks for image classification.For text classification, however, existing synonym substitution based adversar…
Adversarial AttackImage Classificationtext-classificationText ClassificationCertified Robustness to Text Adversarial Attacks by Randomized [MASK]
Recently, few certified defense methods have been developed to provably guarantee the robustness of a text classifier to adversarial synonym substitutions. However, all existing certified defense methods assume that the …
Certified Robustness to Word Substitution Attack with Differential Privacy
The robustness and security of natural language processing (NLP) models are significantly important in real-world applications. In the context of text classification tasks, adversarial examples can be designed by substit…
Adversarial RobustnessClassificationtext-classificationText ClassificationSAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions
State-of-the-art NLP models can often be fooled by human-unaware transformations such as synonymous word substitution. For security reasons, it is of critical importance to develop models with certified robustness that c…
text-classificationText Classification