Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisation
We introduce ordered transfer hyperparameter optimisation (OTHPO), a version of transfer learning for hyperparameter optimisation (HPO) where the tasks follow a sequential order. Unlike for state-of-the-art transfer HPO, the assumption is that each task is most correlated to those immediately before it. This matches many deployed settings, where hyperparameters are retuned as more data is collected; for instance tuning a sequence of movie recommendation systems as more movies and ratings are added. We propose a formal definition, outline the differences to related problems and propose a basic OTHPO method that outperforms state-of-the-art transfer HPO. We empirically show the importance of taking order into account using ten benchmarks. The benchmarks are in the setting of gradually accumulating data, and span XGBoost, random forest, approximate k-nearest neighbor, elastic net, support vector machines and a separate real-world motivated optimisation problem. We open source the benchmarks to foster future research on ordered transfer HPO.
Code (1)
Tasks
Movie RecommendationRecommendation SystemsTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hyperparameter Transfer Learning with Adaptive Complexity
Bayesian optimization (BO) is a sample efficient approach to automatically tune the hyperparameters of machine learning models. In practice, one frequently has to solve similar hyperparameter tuning problems sequentially…
Bayesian OptimizationDecision MakingSequential Decision MakingTransfer LearningTopological Order in Neural Wavefunctions
Topologically ordered states are among the most interesting quantum phases of matter that host emergent quasi-particles having fractional charge and obeying fractional quantum statistics. Theoretical study of such states…
Ordered Kripke Model, Permissibility, and Convergence of Probabilistic Kripke Model
We define a modification of the standard Kripke model, called the ordered Kripke model, by introducing a linear order on the set of accessible states of each state. We first show this model can be used to describe the le…
modelPhase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
Hyperparameter tuning is one of the essential steps to guarantee the convergence of machine learning models. We argue that intuition about the optimal choice of hyperparameters for stochastic gradient descent can be obta…
Enforcing Orderedness to Improve Feature Consistency
Sparse autoencoders (SAEs) have been widely used for interpretability of neural networks, but their learned features often vary across seeds and hyperparameter settings. We introduce Ordered Sparse Autoencoders (OSAE), w…