Learning Prediction Intervals for Model Performance
Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use.
Code (0)
등록된 구현이 없습니다.
Tasks
modelPredictionPrediction IntervalsTransfer LearningSimilar Papers 제목 키워드 기반
RFpredInterval: An R Package for Prediction Intervals with Random Forests and Boosted Forests
Like many predictive models, random forests provide point predictions for new observations. Besides the point prediction, it is important to quantify the uncertainty in the prediction. Prediction intervals provide inform…
PredictionPrediction IntervalsPrediction Intervals in the Beta Autoregressive Moving Average Model
In this paper, we propose five prediction intervals for the beta autoregressive moving average model. This model is suitable for modeling and forecasting variables that assume values in the interval $(0,1)$. Two of the p…
PredictionPrediction IntervalsTight Prediction Intervals Using Expanded Interval Minimization
Prediction intervals are a valuable way of quantifying uncertainty in regression problems. Good prediction intervals should be both correct, containing the actual value between the lower and upper bound at least a target…
PredictionPrediction IntervalsPrediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles
Prediction intervals are a machine- and human-interpretable way to represent predictive uncertainty in a regression analysis. In this paper, we present a method for generating prediction intervals along with point estima…
PredictionPrediction IntervalsConfidence Calibration for Systems with Cascaded Predictive Modules
Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples. However, considering an autonomous system cons…
Conformal PredictionPredictionPrediction Intervalsregression