Stochastic Hyperparameter Optimization through Hypernetworks
Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of weights and hyperparameters. Our process trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our technique converges to locally optimal weights and hyperparameters for sufficiently large hypernetworks. We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.
Code (1)
Tasks
BIG-bench Machine LearningHyperparameter OptimizationStochastic OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stochastic Maximum Likelihood Optimization via Hypernetworks
This work explores maximum likelihood optimization of neural networks through hypernetworks. A hypernetwork initializes the weights of another network, which in turn can be employed for typical functional tasks such as r…
ClassificationGeneral ClassificationregressionQuantum HyperNetworks: Training Binary Neural Networks in Quantum Superposition
Binary neural networks, i.e., neural networks whose parameters and activations are constrained to only two possible values, offer a compelling avenue for the deployment of deep learning models on energy- and memory-limit…
Combinatorial OptimizationHyper-Learning for Gradient-Based Batch Size Adaptation
Scheduling the batch size to increase is an effective strategy to control gradient noise when training deep neural networks. Current approaches implement scheduling heuristics that neglect structure within the optimizati…
Hyperparameter OptimizationSchedulingMagnitude Invariant Parametrizations Improve Hypernetwork Learning
Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunate…
Image GenerationMulti-Task LearningDelta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem. While research on bilevel optimization of neural networks has been dominated by implicit differentiation and u…
Bilevel OptimizationHyperparameter OptimizationImage ClassificationRolling Shutter Correction