Improving Generalization in Semantic Parsing by Increasing Natural Language Variation
Text-to-SQL semantic parsing has made significant progress in recent years, with various models demonstrating impressive performance on the challenging Spider benchmark. However, it has also been shown that these models often struggle to generalize even when faced with small perturbations of previously (accurately) parsed expressions. This is mainly due to the linguistic form of questions in Spider which are overly specific, unnatural, and display limited variation. In this work, we use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations. Existing approaches generate question reformulations either via models trained on Spider or only introduce local changes. In contrast, we leverage the capabilities of large language models to generate more realistic and diverse questions. Using only a few prompts, we achieve a two-fold increase in the number of questions in Spider. Training on this augmented dataset yields substantial improvements on a range of evaluation sets, including robustness benchmarks and out-of-domain data.
Code (1)
Tasks
Data AugmentationSemantic ParsingText to SQLText-To-SQLSimilar Papers 제목 키워드 기반
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?
Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stron…
Semantic ParsingExploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, inc…
Semantic ParsingCalibrated Interpretation: Confidence Estimation in Semantic Parsing
Sequence generation models are increasingly being used to translate natural language into programs, i.e. to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to ex…
Semantic ParsingStructural generalization in COGS: Supertagging is (almost) all you need
In many Natural Language Processing applications, neural networks have been found to fail to generalize on out-of-distribution examples. In particular, several recent semantic parsing datasets have put forward important …
AllSemantic ParsingHow Does Code Pretraining Affect Language Model Task Performance?
Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including…
Language ModelingLanguage ModellingSemantic ParsingWorld Knowledge