Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing
We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic ParsingSimilar Papers 제목 키워드 기반
Defining and Counting Phonological Classes in Cross-linguistic Segment Databases
Recently, there has been an explosion in the availability of large, good-quality cross-linguistic databases such as WALS (Dryer {\&} Haspelmath, 2013), Glottolog (Hammarstrom et al., 2015) and Phoible (Moran {\&} McCloy,…
DiversityOn the Structural Generalization in Text-to-SQL
Exploring the generalization of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous works provided investigations focusing on lexical diversity, including the influenc…
DiversityText to SQLText-To-SQLTackling prediction tasks in relational databases with LLMs
Though large language models (LLMs) have demonstrated exceptional performance across numerous problems, their application to predictive tasks in relational databases remains largely unexplored. In this work, we address t…
PredictionExploring Scaling Laws for EHR Foundation Models
The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, th…
BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Tasks
In the Emotion Recognition in Conversation task, recent investigations have utilized attention mechanisms exploring relationships among utterances from intra- and inter-speakers for modeling emotional interaction between…
Emotion RecognitionEmotion Recognition in Conversation