Zero and Few-shot Semantic Parsing with Ambiguous Inputs
Despite the frequent challenges posed by ambiguity when representing meaning via natural language, it is often ignored or deliberately removed in tasks mapping language to formally-designed representations, which generally assume a one-to-one mapping between linguistic and formal representations. We attempt to address this shortcoming by introducing AmP, a framework, dataset, and challenge for translating ambiguous natural language to formal representations like logic and code. We define templates and generate data for five well-documented linguistic ambiguities. Using AmP, we investigate how several few-shot text-to-code systems handle ambiguity, introducing three new metrics. We find that large pre-trained models perform poorly at capturing the distribution of possible meanings without deliberate instruction. However, models are able to capture the distribution well when ambiguity is attested in their inputs. These results motivate a call for including ambiguity explicitly in datasets and promote considering the distribution of possible outputs when evaluating systems. Data and code: https://github.com/esteng/ambiguous_parsing
Code (1)
Tasks
Semantic ParsingSimilar Papers 제목 키워드 기반
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models
We explore the use of large language models (LLMs) for zero-shot semantic parsing. Semantic parsing involves mapping natural language utterances to task-specific meaning representations. Language models are generally tra…
Extractive Question-AnsweringLanguage ModelingLanguage ModellingQuestion Answering+2SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing.The canonical utterance is often lengthy and complex due to the compositional …
Out-of-Distribution GeneralizationSemantic ParsingSeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing. The canonical utterance is often lengthy and complex due to the compositional…
Out-of-Distribution GeneralizationSemantic ParsingZero-Shot Semantic Parsing for Instructions
We consider a zero-shot semantic parsing task: parsing instructions into compositional logical forms, in domains that were not seen during training. We present a new dataset with 1,390 examples from 7 application domains…
Semantic ParsingTripletHierarchical Neural Data Synthesis for Semantic Parsing
Semantic parsing datasets are expensive to collect. Moreover, even the questions pertinent to a given domain, which are the input of a semantic parsing system, might not be readily available, especially in cross-domain s…
Data AugmentationSemantic ParsingText to SQLText-To-SQL