Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic ParsingSimilar Papers 제목 키워드 기반
Defining and Counting Phonological Classes in Cross-linguistic Segment Databases
Recently, there has been an explosion in the availability of large, good-quality cross-linguistic databases such as WALS (Dryer {\&} Haspelmath, 2013), Glottolog (Hammarstrom et al., 2015) and Phoible (Moran {\&} McCloy,…
DiversityOn the Structural Generalization in Text-to-SQL
Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influenc…
DiversityText to SQLText-To-SQLTackling prediction tasks in relational databases with LLMs
Though large language models (LLMs) have demonstrated exceptional performance across numerous problems, their application to predictive tasks in relational databases remains largely unexplored. In this work, we address t…
PredictionExploring Scaling Laws for EHR Foundation Models
The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, th…
Impact of ECG Dataset Diversity on Generalization of CNN Model for Detecting QRS Complex
Detection of QRS complexes in electrocardiogram (ECG) signal is crucial for automated cardiac diagnosis. Automated QRS detection has been a research topic for over three decades and several of the traditional QRS detecti…
DiversityElectrocardiography (ECG)QRS Complex Detection