Adaptive Uncertainty Quantification for Generative AI
This work is concerned with conformal prediction in contemporary applications (including generative AI) where a black-box model has been trained on data that are not accessible to the user. Mirroring split-conformal inference, we design a wrapper around a black-box algorithm which calibrates conformity scores. This calibration is local and proceeds in two stages by first adaptively partitioning the predictor space into groups and then calibrating sectionally group by group. Adaptive partitioning (self-grouping) is achieved by fitting a robust regression tree to the conformity scores on the calibration set. This new tree variant is designed in such a way that adding a single new observation does not change the tree fit with overwhelmingly large probability. This add-one-in robustness property allows us to conclude a finite sample group-conditional coverage guarantee, a refinement of the marginal guarantee. In addition, unlike traditional split-conformal inference, adaptive splitting and within-group calibration yields adaptive bands which can stretch and shrink locally. We demonstrate benefits of local tightening on several simulated as well as real examples using non-parametric regression. Finally, we consider two contemporary classification applications for obtaining uncertainty quantification around GPT-4o predictions. We conformalize skin disease diagnoses based on self-reported symptoms as well as predicted states of U.S. legislators based on summaries of their ideology. We demonstrate substantial local tightening of the uncertainty sets while attaining similar marginal coverage.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionUncertainty QuantificationSimilar Papers 제목 키워드 기반
Towards Uncertainty Quantification in Generative Model Learning
While generative models have become increasingly prevalent across various domains, fundamental concerns regarding their reliability persist. A crucial yet understudied aspect of these models is the uncertainty quantifica…
Generative Parameter Sampler For Scalable Uncertainty Quantification
Uncertainty quantification has been a core of the statistical machine learning, but its computational bottleneck has been a serious challenge for both Bayesians and frequentists. We propose a model-based framework in qua…
General ClassificationUncertainty QuantificationUncertainty Quantification via Neural Posterior Principal Components
Uncertainty quantification is crucial for the deployment of image restoration models in safety-critical domains, like autonomous driving and biological imaging. To date, methods for uncertainty visualization have mainly …
Autonomous DrivingDenoisingImage DenoisingImage Restoration+4Conformalized Generative Bayesian Imaging: An Uncertainty Quantification Framework for Computational Imaging
Uncertainty quantification plays an important role in achieving trustworthy and reliable learning-based computational imaging. Recent advances in generative modeling and Bayesian neural networks have enabled the developm…
Conformal PredictionImage InpaintingImage ReconstructionUncertainty QuantificationGenerative Conformal Prediction with Vectorized Non-Conformity Scores
Conformal prediction (CP) provides model-agnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings…
Autonomous DrivingConformal PredictionDecision MakingMedical Diagnosis+3