Model Performance Prediction for Hyperparameter Optimization of Deep Learning Models Using High Performance Computing and Quantum Annealing

2023-11-29 · Juan Pablo García Amboage, Eric Wulff, Maria Girone, Tomás F. Pena

Hyperparameter Optimization (HPO) of Deep Learning-based models tends to be a compute resource intensive process as it usually requires to train the target model with many different hyperparameter configurations. We show that integrating model performance prediction with early stopping methods holds great potential to speed up the HPO process of deep learning models. Moreover, we propose a novel algorithm called Swift-Hyperband that can use either classical or quantum support vector regression for performance prediction and benefit from distributed High Performance Computing environments. This algorithm is tested not only for the Machine-Learned Particle Flow model used in High Energy Physics, but also for a wider range of target models from domains such as computer vision and natural language processing. Swift-Hyperband is shown to find comparable (or better) hyperparameters as well as using less computational resources in all test cases.

📄 PDF Abstract BibTeX arXiv:2311.17508

Code (0)

등록된 구현이 없습니다.

Tasks

Deep LearningHyperparameter Optimization

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Early Stopping Early Stopping is a regularization technique for deep neural networks that stops training when parameter updates no longer begin to yield improves on a validation set. In…

HPO In machine learning, a hyperparameter is a parameter whose value is used to control learning process, and HPO is the problem of choosing a set of optimal hyperparameters for a…

The Role of Hyperparameters in Predictive Multiplicity

2025-03-13 · Mustafa Cavus, Katarzyna Woźnica, Przemysław Biecek

This paper investigates the critical role of hyperparameters in predictive multiplicity, where different machine learning models trained on the same dataset yield divergent predictions for identical inputs. These inconsi…

Fairness Hyperparameter Optimization Prediction

Accelerating Neural Architecture Search using Performance Prediction

2017-05-30 · ICLR 2018 1 · Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik

Methods for neural network hyperparameter optimization and meta-modeling are computationally expensive due to the need to train a large number of model configurations. In this paper, we show that standard frequentist reg…

Hyperparameter Optimization Language Modeling Language Modelling Neural Architecture Search+4

Beyond algorithm hyperparameters: on preprocessing hyperparameters and associated pitfalls in machine learning applications

2024-12-04 · Christina Sauer, Anne-Laure Boulesteix, Luzia Hanßum, Farina Hodiamont 외

Adequately generating and evaluating prediction models based on supervised machine learning (ML) is often challenging, especially for less experienced users in applied research areas. Special attention is required in set…

Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

2019-12-11 · Daniel T. Chang

Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consumi…

Bayesian Optimization Gaussian Processes Graph Classification Hyperparameter Optimization

Which Hyperparameters to Optimise? An Investigation of Evolutionary Hyperparameter Optimisation in Graph Neural Network For Molecular Property Prediction

2021-04-13 · Yingfang Yuan, Wenjun Wang, Wei Pang

Recently, the study of graph neural network (GNN) has attracted much attention and achieved promising performance in molecular property prediction. Most GNNs for molecular property prediction are proposed based on the id…

Graph Neural Network Hyperparameter Optimization Molecular Property Prediction Prediction+1