Distribution-free binary classification: prediction sets, confidence intervals and calibration
We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting, that is without making any distributional assumptions on the data. With a focus towards calibration, we establish a 'tripod' of theorems that connect these three notions for score-based classifiers. A direct implication is that distribution-free calibration is only possible, even asymptotically, using a scoring function whose level sets partition the feature space into at most countably many sets. Parametric calibration schemes such as variants of Platt scaling do not satisfy this requirement, while nonparametric schemes based on binning do. To close the loop, we derive distribution-free confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration. We also derive extensions to settings with streaming data and covariate shift.
Code (1)
Tasks
Binary ClassificationClassificationGeneral ClassificationUncertainty QuantificationSimilar Papers 제목 키워드 기반
Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction
Ensuring model calibration is critical for reliable predictions, yet popular distribution-free methods, such as histogram binning and isotonic regression, provide only asymptotic guarantees. We introduce a unified framew…
Binary ClassificationConformal PredictionPredictionPrediction IntervalsDistribution-free Conformal Prediction for Ordinal Classification
Conformal prediction is a general distribution-free approach for constructing prediction sets combined with any machine learning algorithm that achieve valid marginal or conditional coverage in finite samples. Ordinal cl…
ClassificationConformal PredictionOrdinal ClassificationPrediction+1Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings
In this paper we suggest two statistical hypothesis tests for the regression function of binary classification based on conditional kernel mean embeddings. The regression function is a fundamental object in classificatio…
Binary ClassificationClassificationGeneral ClassificationregressionDistribution-Free Inference for the Regression Function of Binary Classification
One of the key objects of binary classification is the regression function, i.e., the conditional expectation of the class labels given the inputs. With the regression function not only a Bayes optimal classifier can be …
Binary ClassificationregressionDistribution-free uncertainty quantification for classification under label shift
Trustworthy deployment of ML models requires a proper measure of uncertainty, especially in safety-critical applications. We focus on uncertainty quantification (UQ) for classification problems via two avenues -- predict…
ClassificationConformal PredictionGeneral ClassificationPrediction+1