Blocked Cross-Validation: A Precise and Efficient Method for Hyperparameter Tuning
Hyperparameter tuning plays a crucial role in optimizing the performance of predictive learners. Cross--validation (CV) is a widely adopted technique for estimating the error of different hyperparameter settings. Repeated cross-validation (RCV) has been commonly employed to reduce the variability of CV errors. In this paper, we introduce a novel approach called blocked cross-validation (BCV), where the repetitions are blocked with respect to both CV partition and the random behavior of the learner. Theoretical analysis and empirical experiments demonstrate that BCV provides more precise error estimates compared to RCV, even with a significantly reduced number of runs. We present extensive examples using real--world data sets to showcase the effectiveness and efficiency of BCV in hyperparameter tuning. Our results indicate that BCV outperforms RCV in hyperparameter tuning, achieving greater precision with fewer computations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Nested cross-validation when selecting classifiers is overzealous for most practical applications
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The u…
On hyperparameter tuning in general clustering problemsm
Tuning hyperparameters for unsupervised learning problems is difficult in general due to the lack of ground truth for validation. However, the success of most clustering methods depends heavily on the correct choice of t…
ClusteringCommunity DetectionModel SelectionPerformance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data
Machine-learning algorithms have gained popularity in recent years in the field of ecological modeling due to their promising results in predictive performance of classification problems. While the application of such al…
Additive modelsBIG-bench Machine LearningregressionOne For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer
Multilingual language models enable zero-shot cross-lingual transfer (ZS-XLT): fine-tuned on sizable source-language task data, they perform the task in target languages without labeled instances. The effectiveness of ZS…
AllCross-Lingual TransferModel SelectionNER+3Model Validation Using Mutated Training Labels: An Exploratory Study
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, retrains the model against the mutated data…
BIG-bench Machine LearningGeneral ClassificationModel Selection