SOAK: Same/Other/All K-fold cross-validation for estimating similarity of patterns in data subsets
In many real-world applications of machine learning, we are interested to know if it is possible to train on the data that we have gathered so far, and obtain accurate predictions on a new test data subset that is qualitatively different in some respect (time period, geographic region, etc). Another question is whether data subsets are similar enough so that it is beneficial to combine subsets during model training. We propose SOAK, Same/Other/All K-fold cross-validation, a new method which can be used to answer both questions. SOAK systematically compares models which are trained on different subsets of data, and then used for prediction on a fixed test subset, to estimate the similarity of learnable/predictable patterns in data subsets. We show results of using SOAK on six new real data sets (with geographic/temporal subsets, to check if predictions are accurate on new subsets), 3 image pair data sets (subsets are different image types, to check that we get smaller prediction error on similar images), and 11 benchmark data sets with predefined train/test splits (to check similarity of predefined splits).
Code (1)
Tasks
AllSimilar Papers 제목 키워드 기반
Effect of IBA concentration and water soaking on rooting hardwood cuttings of black mulberry (Morus nigra L.)
The research was conducted at the College of Agricultural Sciences Engineering/University of Sulaimani/ Kurdistan Region-Iraqi to investigate effects of different concentrations of IBA (0, 3000, 4000 and 5000 ppm) and so…
Improving Efficiency of SVM k-fold Cross-validation by Alpha Seeding
The k-fold cross-validation is commonly used to evaluate the effectiveness of SVMs with the selected hyper-parameters. It is known that the SVM k-fold cross-validation is expensive, since it requires training k SVMs. How…
Efficient Cross-Validation of Echo State Networks
Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a sing…
One-Shot LearningTime SeriesTime Series AnalysisFast calculation of Gaussian Process multiple-fold cross-validation residuals and their covariances
We generalize fast Gaussian process leave-one-out formulae to multiple-fold cross-validation, highlighting in turn the covariance structure of cross-validation residuals in both Simple and Universal Kriging frameworks. W…
Stability Regularized Cross-Validation
We revisit the problem of ensuring strong test-set performance via cross-validation. Motivated by the generalization theory literature, we propose a nested k-fold cross-validation scheme that selects hyperparameters by m…
regression