Semi-Supervised Risk Control via Prediction-Powered Inference
The risk-controlling prediction sets (RCPS) framework is a general tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. The key idea behind this framework is to use labeled hold-out calibration data to tune a hyper-parameter that affects the error rate of the resulting prediction rule. However, the limitation of such a calibration scheme is that with limited hold-out data, the tuned hyper-parameter becomes noisy and leads to a prediction rule with an error rate that is often unnecessarily conservative. To overcome this sample-size barrier, we introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper-parameter without compromising statistical validity. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks. We demonstrate the benefits and validity of our proposal through two real-data experiments: few-shot image classification and early time series classification.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot Image Classificationimage-classificationImage ClassificationPredictionTime Series ClassificationSimilar Papers 제목 키워드 기반
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach b…
Image ClassificationSemi-Supervised Treatment Effect Estimation with Unlabeled Covariates for Prediction-Powered Causal Inference
This study investigates treatment effect estimation in the semi-supervised setting, also can be interpreted as prediction-powered inference. In our setting, we can use not only the standard triple of covariates, treatmen…
Causal InferencePPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization
We study semi-supervised stochastic optimization when labeled data is scarce but predictions from pre-trained models are available. PPI and SVRG both reduce variance through control variates -- PPI uses predictions, SVRG…
Stochastic OptimizationPrediction-powered Inference by Mixture of Experts
The rapidly expanding artificial intelligence (AI) industry has produced diverse yet powerful prediction tools, each with its own network architecture, training strategy, data-processing pipeline, and domain-specific str…
A Comparison of Self-Supervised Pretraining Approaches for Predicting Disease Risk from Chest Radiograph Images
Deep learning is the state-of-the-art for medical imaging tasks, but requires large, labeled datasets. For risk prediction, large datasets are rare since they require both imaging and follow-up (e.g., diagnosis codes). H…
Diagnosticimage-classificationImage Classificationobject-detection+4