When Can We Reuse a Calibration Set for Multiple Conformal Predictions?

2025-06-24 · A. A. Balinsky, A. D. Balinsky

Reliable uncertainty quantification is crucial for the trustworthiness of machine learning applications. Inductive Conformal Prediction (ICP) offers a distribution-free framework for generating prediction sets or intervals with user-specified confidence. However, standard ICP guarantees are marginal and typically require a fresh calibration set for each new prediction to maintain their validity. This paper addresses this practical limitation by demonstrating how e-conformal prediction, in conjunction with Hoeffding's inequality, can enable the repeated use of a single calibration set with a high probability of preserving the desired coverage. Through a case study on the CIFAR-10 dataset, we train a deep neural network and utilise a calibration set to estimate a Hoeffding correction. This correction allows us to apply a modified Markov's inequality, leading to the construction of prediction sets with quantifiable confidence. Our results illustrate the feasibility of maintaining provable performance in conformal prediction while enhancing its practicality by reducing the need for repeated calibration. The code for this work is publicly available.

📄 PDF Abstract BibTeX arXiv:2506.19689

Code (0)

등록된 구현이 없습니다.

Tasks

Conformal PredictionPredictionUncertainty Quantification

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees

2026-02-25 · Vagner Santos, Victor Coscrato, Luben Cabezas, Rafael Izbicki 외 arxiv

Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty. Conformal prediction provides distribution-free marginal c…

Does confidence calibration improve conformal prediction?

2024-02-06 · Huajun Xi, Jianguo Huang, Kangdao Liu, Lei Feng 외

Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Previous works often employ temperature sca…

Conformal Prediction Prediction Uncertainty Quantification

Adaptive Multi-Scale Forecasting and Gate-Localized Conformal Prediction for Multivariate Nonstationary Time Series

2026-07-25 · Ziling Ma, Junshu Jiang, Ángel López-Oriona, Ying Sun 외 arxiv

We propose ABF-T-GLCP, a model-agnostic framework for forecasting and uncertainty quantification in nonstationary multivariate time series. The central idea is to learn an adaptive predictive state representation for poi…

Conformalized Interval Arithmetic with Symmetric Calibration

2024-08-20 · Rui Luo, Zhixin Zhou

Uncertainty quantification is essential in decision-making, especially when joint distributions of random variables are involved. While conformal prediction provides distribution-free prediction sets with valid coverage …

Conformal Prediction Decision Making Prediction Prediction Intervals+2

Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples

2022-10-09 · Fatih Furkan Yilmaz, Reinhard Heckel

Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a use…

Conformal Prediction Prediction Intervals