HyperSTAR: Task-Aware Hyperparameters for Deep Networks

2020-05-21 · CVPR 2020 6 · Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso, Mei Chen, Yun Fu

While deep neural networks excel in solving visual recognition tasks, they require significant effort to find hyperparameters that make them work optimally. Hyperparameter Optimization (HPO) approaches have automated the process of finding good hyperparameters but they do not adapt to a given task (task-agnostic), making them computationally inefficient. To reduce HPO time, we present HyperSTAR (System for Task Aware Hyperparameter Recommendation), a task-aware method to warm-start HPO for deep neural networks. HyperSTAR ranks and recommends hyperparameters by predicting their performance conditioned on a joint dataset-hyperparameter space. It learns a dataset (task) representation along with the performance predictor directly from raw images in an end-to-end fashion. The recommendations, when integrated with an existing HPO method, make it task-aware and significantly reduce the time to achieve optimal performance. We conduct extensive experiments on 10 publicly available large-scale image classification datasets over two different network architectures, validating that HyperSTAR evaluates 50% less configurations to achieve the best performance compared to existing methods. We further demonstrate that HyperSTAR makes Hyperband (HB) task-aware, achieving the optimal accuracy in just 25% of the budget required by both vanilla HB and Bayesian Optimized HB~(BOHB).

📄 PDF Abstract BibTeX arXiv:2005.10524

Code (0)

등록된 구현이 없습니다.

Tasks

Hyperparameter Optimization image-classification Image Classification

Methods 이 논문이 사용한 방법론

HPO In machine learning, a hyperparameter is a parameter whose value is used to control learning process, and HPO is the problem of choosing a set of optimal hyperparameters for a…

LLM can Achieve Self-Regulation via Hyperparameter Aware Generation

2024-02-17 · Siyin Wang, ShiMin Li, Tianxiang Sun, Jinlan Fu 외

In the realm of Large Language Models (LLMs), users commonly employ diverse decoding strategies and adjust hyperparameters to control the generated text. However, a critical question emerges: Are LLMs conscious of the ex…

Text Generation

Meta-Learning with Adaptive Hyperparameters

2020-10-31 · NeurIPS 2020 12 · Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim 외

Despite its popularity, several recent works question the effectiveness of MAML when test tasks are different from training tasks, thus suggesting various task-conditioned methodology to improve the initialization. Inste…

Few-Shot Learning Meta-Learning

SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees

2023-07-10 · Aleksei Sorokin, Xinran Zhu, Eric Hans Lee, Bolong Cheng

Gradient boosted trees (GBTs) are ubiquitous models used by researchers, machine learning (ML) practitioners, and data scientists because of their robust performance, interpretable behavior, and ease-of-use. One critical…

AutoML Hyperparameter Optimization

A Framework for History-Aware Hyperparameter Optimisation in Reinforcement Learning

2023-03-09 · Juan Marcelo Parra-Ullauri, Chen Zhen, Antonio García-Domínguez, Nelly Bencomo 외

A Reinforcement Learning (RL) system depends on a set of initial conditions (hyperparameters) that affect the system's performance. However, defining a good choice of hyperparameters is a challenging problem. Hyperparame…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Principled Architecture-aware Scaling of Hyperparameters

2024-02-27 · Wuyang Chen, Junru Wu, Zhangyang Wang, Boris Hanin

Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, s…

AutoML