Using a thousand optimization tasks to learn hyperparameter search strategies
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as ~29 million training curves for these problems and the corresponding hyperparameters.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityGeneral Classificationimage-classificationImage ClassificationLanguage ModelingLanguage ModellingMeta-LearningSimilar Papers 제목 키워드 기반
TaskSet: A Dataset of Optimization Tasks
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected o…
Diversityimage-classificationImage ClassificationLanguage Modeling+2Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm
Modern machine learning algorithms usually involve tuning multiple (from one to thousands) hyperparameters which play a pivotal role in terms of model generalizability. Black-box optimization and gradient-based algorithm…
Hyperparameter OptimizationStochastic Hyperparameter Optimization through Hypernetworks
Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of wei…
BIG-bench Machine LearningHyperparameter OptimizationStochastic OptimizationEfficient hyperparameter optimization by way of PAC-Bayes bound minimization
Identifying optimal values for a high-dimensional set of hyperparameters is a problem that has received growing attention given its importance to large-scale machine learning applications such as neural architecture sear…
Hyperparameter OptimizationNeural Architecture SearchDeepTraffic: Crowdsourced Hyperparameter Tuning of Deep Reinforcement Learning Systems for Multi-Agent Dense Traffic Navigation
We present a traffic simulation named DeepTraffic where the planning systems for a subset of the vehicles are handled by a neural network as part of a model-free, off-policy reinforcement learning process. The primary go…
Autonomous DrivingAutonomous NavigationDeep Reinforcement LearningQ-Learning+3