Confidence Modeling for Neural Semantic Parsing
In this work we focus on confidence modeling for neural semantic parsers which are built upon sequence-to-sequence models. We outline three major causes of uncertainty, and design various metrics to quantify these factors. These metrics are then used to estimate confidence scores that indicate whether model predictions are likely to be correct. Beyond confidence estimation, we identify which parts of the input contribute to uncertain predictions allowing users to interpret their model, and verify or refine its input. Experimental results show that our confidence model significantly outperforms a widely used method that relies on posterior probability, and improves the quality of interpretation compared to simply relying on attention scores.
Code (1)
Tasks
Semantic ParsingSimilar Papers 제목 키워드 기반
Calibrated Interpretation: Confidence Estimation in Semantic Parsing
Sequence generation models are increasingly being used to translate natural language into programs, i.e. to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to ex…
Semantic ParsingDid You Mean...? Confidence-based Trade-offs in Semantic Parsing
We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost w…
Semantic ParsingModel-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the results. In this paper, we propose a new, unified formulation of the interactive semant…
Semantic ParsingText to SQLText-To-SQLRenovating Parsing R-CNN for Accurate Multiple Human Parsing
Multiple human parsing aims to segment various human parts and associate each part with the corresponding instance simultaneously. This is a very challenging task due to the diverse human appearance, semantic ambiguity o…
Human ParsingUncertainty and Traffic-Aware Active Learning for Semantic Parsing
Collecting training data for semantic parsing is a time-consuming and expensive task. As a result, there is growing interest in industry to reduce the number of annotations required to train a semantic parser, both to cu…
Active LearningSemantic Parsing