AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries
Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.
Code (1)
Tasks
Text to SQLText-To-SQLSimilar Papers 제목 키워드 기반
Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
Given a text description, most existing semantic parsers synthesize a program in one shot. However, it is quite challenging to produce a correct program solely based on the description, which in reality is often ambiguou…
Hierarchical Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Semantic Parsing of Ambiguous Input through Paraphrasing and Verification
We propose a new method for semantic parsing of ambiguous and ungrammatical input, such as search queries. We do so by building on an existing semantic parsing framework that uses synchronous context free grammars (SCFG)…
Language ModelingLanguage ModellingSemantic ParsingSentenceSynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA
Text-to-SQL parsing and end-to-end question answering (E2E TQA) are two main approaches for Table-based Question Answering task. Despite success on multiple benchmarks, they have yet to be compared and their synergy rema…
Answer SelectionQuestion AnsweringSemantic ParsingSQL Parsing+3Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions
Many open-domain questions are under-specified and thus have multiple possible answers, each of which is correct under a different interpretation of the question. Answering such ambiguous questions is challenging, as it …
Passage RetrievalQuestion AnsweringRetrievalCQR-SQL: Conversational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parsers
Context-dependent text-to-SQL is the task of translating multi-turn questions into database-related SQL queries. Existing methods typically focus on making full use of history context or previously predicted SQL for curr…
SQL ParsingText to SQLText-To-SQL