ACHO: Adaptive Conformal Hyperparameter Optimization
Several novel frameworks for hyperparameter search have emerged in the last decade, but most rely on strict, often normal, distributional assumptions, limiting search model flexibility. This paper proposes a novel optimization framework based on upper confidence bound sampling of conformal confidence intervals, whose weaker assumption of exchangeability enables greater choice of search model architectures. Several such architectures were explored and benchmarked on hyperparameter search of random forests and convolutional neural networks, displaying satisfactory interval coverage and superior tuning performance to random search.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionHyperparameter OptimizationSimilar Papers 제목 키워드 기반
Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
The variability and biases in the real-world performance benchmarking of deep learning models for medical imaging compromise their trustworthiness for real-world deployment. The common approach of holding out a single fi…
BenchmarkingHyperparameter OptimizationSpeedCP: Fast Kernel-based Conditional Conformal Prediction
Conformal prediction provides distribution-free prediction sets with finite-sample conditional guarantees. We build upon the RKHS-based framework of Gibbs et al. (2023), which leverages families of covariate shifts to pr…
Temporal Conformal Prediction (TCP): A Distribution-Free Statistical and Machine Learning Framework for Adaptive Risk Forecasting
We propose \textbf{Temporal Conformal Prediction (TCP)}, a distribution-free framework for constructing well-calibrated prediction intervals in nonstationary time series. TCP couples a modern quantile forecaster with a r…
Enhancing Performance and Calibration in Quantile Hyperparameter Optimization
Bayesian hyperparameter optimization relies heavily on Gaussian Process (GP) surrogates, due to robust distributional posteriors and strong performance on limited training samples. GPs however underperform in categorical…
Hyperparameter OptimizationEfficient Optimization of Echo State Networks for Time Series Datasets
Echo State Networks (ESNs) are recurrent neural networks that only train their output layer, thereby precluding the need to backpropagate gradients through time, which leads to significant computational gains. Neverthele…
AstronomyBayesian OptimizationTime SeriesTime Series Analysis