Non-Exchangeable Conformal Language Generation with Nearest Neighbors
Quantifying uncertainty in automatically generated text is important for letting humans check potential hallucinations and making systems more reliable. Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i.i.d. assumptions are not realistic. In this paper, we bridge this gap by leveraging recent results on non-exchangeable conformal prediction, which still ensures bounds on coverage. The result, non-exchangeable conformal nucleus sampling, is a novel extension of the conformal prediction framework to generation based on nearest neighbors. Our method can be used post-hoc for an arbitrary model without extra training and supplies token-level, calibrated prediction sets equipped with statistical guarantees. Experiments in machine translation and language modeling show encouraging results in generation quality. By also producing tighter prediction sets with good coverage, we thus give a more theoretically principled way to perform sampling with conformal guarantees.
Code (1)
Tasks
Conformal PredictionLanguage ModelingLanguage ModellingMachine TranslationPredictionText GenerationSimilar Papers 제목 키워드 기반
Applying Regression Conformal Prediction with Nearest Neighbors to time series data
In this paper, we apply conformal prediction to time series data. Conformal prediction isa method that produces predictive regions given a confidence level. The regions outputs arealways valid under the exchangeability a…
Conformal PredictionPredictionPrediction Intervalsregression+3Extending Conformal Prediction to Hidden Markov Models with Exact Validity via de Finetti's Theorem for Markov Chains
Conformal prediction is a widely used method to quantify the uncertainty of a classifier under the assumption of exchangeability (e.g., IID data). We generalize conformal prediction to the Hidden Markov Model (HMM) frame…
Conformal PredictionPredictionvalidBeyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors
Conformal prediction (CP) is a robust framework for distribution-free uncertainty quantification, but it requires exchangeable data to ensure valid prediction sets at a user-specified significance level. When this assump…
Computational EfficiencyConformal PredictionPredictionUncertainty Quantification+1Non-Exchangeable Conformal Risk Control
Split conformal prediction has recently sparked great interest due to its ability to provide formally guaranteed uncertainty sets or intervals for predictions made by black-box neural models, ensuring a predefined probab…
Conformal PredictionTime SeriesConformal Prediction for Stochastic Decision-Making of PV Power in Electricity Markets
This paper studies the use of conformal prediction (CP), an emerging probabilistic forecasting method, for day-ahead photovoltaic power predictions to enhance participation in electricity markets. First, machine learning…
Conformal PredictionDecision MakingUncertainty Quantification