AutoCP: Automated Pipelines for Accurate Prediction Intervals
Successful application of machine learning models to real-world prediction problems, e.g. financial forecasting and personalized medicine, has proved to be challenging, because such settings require limiting and quantifying the uncertainty in the model predictions, i.e. providing valid and accurate prediction intervals. Conformal Prediction is a distribution-free approach to construct valid prediction intervals in finite samples. However, the prediction intervals constructed by Conformal Prediction are often (because of over-fitting, inappropriate measures of nonconformity, or other issues) overly conservative and hence inadequate for the application(s) at hand. This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP). Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate while optimizing the interval length to be accurate and less conservative. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
AutoMLBIG-bench Machine LearningConformal PredictionPredictionPrediction IntervalsSelf-Driving CarsvalidSimilar Papers 제목 키워드 기반
Imputation of missing sub-hourly precipitation data in a large sensor network: a machine learning approach
Precipitation data collected at sub-hourly resolution represents specific challenges for missing data recovery by being largely stochastic in nature and highly unbalanced in the duration of rain vs non-rain. Here we pres…
BIG-bench Machine LearningImputationMissing ValuesEfficient Conformal Volumetry for Template-Based Segmentation
Template-based segmentation, a widely used paradigm in medical imaging, propagates anatomical labels via deformable registration from a labeled atlas to a target image, and is often used to compute volumetric biomarkers …
Learning Prediction Intervals for Model Performance
Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality asses…
modelPredictionPrediction IntervalsTransfer LearningConRad: Efficient Conformal Prediction for Radiomics
Radiomic features derived from medical images and segmentation masks are used to support decision making in clinical imaging pipelines. In practice, these features are often computed from predicted masks, but segmentatio…
Decision MakingAutomated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline
Multiple Deep Neural Networks (DNNs) integrated into single Deep Learning (DL) inference pipelines e.g. Multi-Task Learning (MTL) or Ensemble Learning (EL), etc., albeit very accurate, pose challenges for edge deployment…
Ensemble LearningMulti-Task LearningQuantization