Generalization and Informativeness of Conformal Prediction
The safe integration of machine learning modules in decision-making processes hinges on their ability to quantify uncertainty. A popular technique to achieve this goal is conformal prediction (CP), which transforms an arbitrary base predictor into a set predictor with coverage guarantees. While CP certifies the predicted set to contain the target quantity with a user-defined tolerance, it does not provide control over the average size of the predicted sets, i.e., over the informativeness of the prediction. In this work, a theoretical connection is established between the generalization properties of the base predictor and the informativeness of the resulting CP prediction sets. To this end, an upper bound is derived on the expected size of the CP set predictor that builds on generalization error bounds for the base predictor. The derived upper bound provides insights into the dependence of the average size of the CP set predictor on the amount of calibration data, the target reliability, and the generalization performance of the base predictor. The theoretical insights are validated using simple numerical regression and classification tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionDecision MakingInformativenessPredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalization and Informativeness of Weighted Conformal Risk Control Under Covariate Shift
Predictive models are often required to produce reliable predictions under statistical conditions that are not matched to the training data. A common type of training-testing mismatch is covariate shift, where the condit…
InformativenessPredictionvalidInformativeness of Weighted Conformal Prediction
Weighted conformal prediction (WCP), a recently proposed framework, provides uncertainty quantification with the flexibility to accommodate different covariate distributions between training and test data. However, it is…
Conformal PredictionInformativenessPredictionPrediction Intervals+1The Penalized Inverse Probability Measure for Conformal Classification
The deployment of safe and trustworthy machine learning systems, and particularly complex black box neural networks, in real-world applications requires reliable and certified guarantees on their performance. The conform…
ClassificationConformal Predictionimage-classificationImage Classification+1Non-Asymptotic Analysis of Efficiency in Conformalized Regression
Conformal prediction provides prediction sets with coverage guarantees. The informativeness of conformal prediction depends on its efficiency, typically quantified by the expected size of the prediction set. Prior work o…
Weighted Aggregation of Conformity Scores for Classification
Conformal prediction is a powerful framework for constructing prediction sets with valid coverage guarantees in multi-class classification. However, existing methods often rely on a single score function, which can limit…
ClassificationConformal PredictionInformativenessMulti-class Classification+2