Non-greedy Gradient-based Hyperparameter Optimization Over Long Horizons
Gradient-based meta-learning has earned a widespread popularity in few-shot learning, but remains broadly impractical for tasks with long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn meta-parameters online, but this introduces greediness which comes with a significant performance drop. In this work, we enable non-greedy meta-learning of hyperparameters over long horizons by sharing hyperparameters that are contiguous in time, and using the sign of hypergradients rather than their magnitude to indicate convergence. We implement this with forward-mode differentiation, which we extend to the popular momentum-based SGD optimizer. We demonstrate that the hyperparameters of this optimizer can be learned non-greedily without gradient degradation over $\sim 10^4$ inner gradient steps, by only requiring $\sim 10$ outer gradient steps. On CIFAR-10, we outperform greedy and random search methods for the same computational budget by nearly $10\%$. Code will be available upon publication.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningHyperparameter OptimizationMeta-LearningSimilar Papers 제목 키워드 기반
Gradient-based Hyperparameter Optimization Over Long Horizons
Gradient-based hyperparameter optimization has earned a widespread popularity in the context of few-shot meta-learning, but remains broadly impractical for tasks with long horizons (many gradient steps), due to memory sc…
Hyperparameter OptimizationMeta-LearningBayesian Hyperparameter Optimization for Ensemble Learning
In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-s…
Bayesian OptimizationEnsemble LearningHyperparameter OptimizationGreedy Alignment Principle for Optimizer Selection
Recent works have shown that gradient-update alignment is a powerful signal for modulating optimizer updates, often leading to faster training. We promote this update-wise heuristic as a mathematically grounded principle…
Image ClassificationEvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing a…
Cross-Domain Few-Shotcross-domain few-shot learningFew-Shot LearningHyperparameter Optimization+1Improving Fast Minimum-Norm Attacks with Hyperparameter Optimization
Evaluating the adversarial robustness of machine learning models using gradient-based attacks is challenging. In this work, we show that hyperparameter optimization can improve fast minimum-norm attacks by automating the…
Adversarial RobustnessHyperparameter Optimization