Transfer Learning for Bayesian HPO with End-to-End Meta-Features

2021-09-29 · Hadi Samer Jomaa, Sebastian Pineda Arango, Lars Schmidt-Thieme, Josif Grabocka

Hyperparameter optimization (HPO) is a crucial component of deploying machine learning models, however, it remains an open problem due to the resource-constrained number of possible hyperparameter evaluations. As a result, prior work focus on exploring the direction of transfer learning for tackling the sample inefficiency of HPO. In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efficiently on a new (unseen) target task. We design DKLM to capture the similarity between hyperparameter configurations with an end-to-end meta-feature network that embeds the set of evaluated configurations and their respective performance. As a result, our novel DKLM can learn contextualized dataset-specific similarity representations for hyperparameter configurations. We experimentally validate the performance of DKLM in a wide range of HPO meta-datasets from OpenML and demonstrate the empirical superiority of our method against a series of state-of-the-art baselines.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Hyperparameter Optimization Transfer Learning

Methods 이 논문이 사용한 방법론

HPO In machine learning, a hyperparameter is a parameter whose value is used to control learning process, and HPO is the problem of choosing a set of optimal hyperparameters for a…

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

Boundedly Rational Meta-Learning in Sequential Consumer Choice

2026-05-15 · Mehrzad Khosravi, Max Kleiman-Weiner, Hema Yoganarasimhan arxiv

Many consumer decisions are repeated choices under uncertainty. Standard models capture these decisions using Bayesian learning and dynamic programming: consumers update beliefs from feedback and use those beliefs to gui…

Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization

2020-11-04 · Sharu Theresa Jose, Osvaldo Simeone, Giuseppe Durisi

Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, suc…

Inductive Bias Meta-Learning

Transfer Bayesian Meta-learning via Weighted Free Energy Minimization

2021-06-20 · Yunchuan Zhang, Sharu Theresa Jose, Osvaldo Simeone

Meta-learning optimizes the hyperparameters of a training procedure, such as its initialization, kernel, or learning rate, based on data sampled from a number of auxiliary tasks. A key underlying assumption is that the a…

Gaussian Processes Meta-Learning regression

Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels

2019-10-11 · NeurIPS 2020 12 · Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O'Boyle 외

Recently, different machine learning methods have been introduced to tackle the challenging few-shot learning scenario that is, learning from a small labeled dataset related to a specific task. Common approaches have tak…

Bayesian Inference Domain Adaptation Few-Shot Image Classification Few-Shot Learning+3

Probabilistic Meta-Learning for Bayesian Optimization

2021-01-01 · Felix Berkenkamp, Anna Eivazi, Lukas Grossberger, Kathrin Skubch 외

Transfer and meta-learning algorithms leverage evaluations on related tasks in order to significantly speed up learning or optimization on a new problem. For applications that depend on uncertainty estimates, e.g., in Ba…

Bayesian Optimization Meta-Learning Transfer Learning