Online Selective Conformal Prediction: Errors and Solutions
In online selective conformal inference, data arrives sequentially, and prediction intervals are constructed only when an online selection rule is met. Since online selections may break the exchangeability between the selected test datum and the rest of the data, one must correct for this by suitably selecting the calibration data. In this paper, we evaluate existing calibration selection strategies and pinpoint some fundamental errors in the associated claims that guarantee selection-conditional coverage and control of the false coverage rate (FCR). To address these shortcomings, we propose novel calibration selection strategies that provably preserve the exchangeability of the calibration data and the selected test datum. Consequently, we demonstrate that online selective conformal inference with these strategies guarantees both selection-conditional coverage and FCR control. Our theoretical findings are supported by experimental evidence examining tradeoffs between valid methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionPredictionPrediction IntervalsvalidSimilar Papers 제목 키워드 기반
Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach
Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any s…
Drug DiscoveryDenoised Conformal Alignment for Reliable Selection of Conditional Average Treatment Effect Predictions
In selective deployment, practitioners act only on a model-chosen subset of individuals based on predicted conditional average treatment effects, but marginal conformal guarantees need not control reliability on that sel…
Online selective conformal inference: adaptive scores, convergence rate and optimality
In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of \cite{gibbs2021adaptive}. For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set …
Conformal Online Learning of Deep Koopman Linear Embeddings
We introduce Conformal Online Learning of Koopman embeddings (COLoKe), a novel framework for adaptively updating Koopman-invariant representations of nonlinear dynamical systems from streaming data. Our modeling approach…
Robust Online Conformal Prediction under Uniform Label Noise
Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Recent work develops online conformal predi…
Conformal PredictionPredictionUncertainty Quantification