Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities
Calibration is crucial as large language models (LLMs) are increasingly deployed to convert natural language queries into SQL for commercial databases. In this work, we investigate calibration techniques for assigning confidence to generated SQL queries. We show that a straightforward baseline -- deriving confidence from the model's full-sequence probability -- outperforms recent methods that rely on follow-up prompts for self-checking and confidence verbalization. Our comprehensive evaluation, conducted across two widely-used Text-to-SQL benchmarks and multiple LLM architectures, provides valuable insights into the effectiveness of various calibration strategies.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language QueriesText to SQLText-To-SQLSimilar Papers 제목 키워드 기반
Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers
Modern NLP classifiers are known to return uncalibrated estimations of class posteriors. Existing methods for posterior calibration rescale the predicted probabilities but often have an adverse impact on final classifica…
ClassificationPosterior Calibrated Training on Sentence Classification Tasks
Most classification models work by first predicting a posterior probability distribution over all classes and then selecting that class with the largest estimated probability. In many settings however, the quality of pos…
ClassificationGeneral ClassificationSentenceSentence ClassificationCut your Losses with Squentropy
Nearly all practical neural models for classification are trained using cross-entropy loss. Yet this ubiquitous choice is supported by little theoretical or empirical evidence. Recent work (Hui & Belkin, 2020) suggests t…
ClassificationMulti-class ClassificationImproved Uncertainty Post-Calibration via Rank Preserving Transforms
Modern machine learning models with high accuracy often exhibit poor uncertainty calibration: the output probabilities of the model do not reflect its accuracy, and tend to be over-confident. Existing post-calibration me…
text-classificationText ClassificationMitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment
TabPFN has recently gained attention as a foundation model for tabular datasets, achieving strong performance by leveraging in-context learning on synthetic data. However, we find that TabPFN is vulnerable to label shift…