Gradient-based Hyperparameter Optimization without Validation Data for Learning fom Limited Labels
Optimizing hyperparameters of machine learning algorithms especially for limited labeled data is important but difficult, because then obtaining enough validation data is practically impossible. Bayesian model selection enables hyperparameter optimization \emph{without validation data}, but it requires Hessian log determinants, which is computationally demanding for deep neural networks. We study methods to efficiently approximate Hessian log determinants and empirically demonstrate that approximated Bayesian model selection can effectively tune hyperparameters of algorithms of deep semi-supervised learning and learning from noisy labels.
Code (0)
등록된 구현이 없습니다.
Tasks
Hyperparameter OptimizationModel SelectionSimilar Papers 제목 키워드 기반
Optimizing for Generalization in Machine Learning with Cross-Validation Gradients
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that th…
BIG-bench Machine LearningHyperparameter OptimizationregressionOptimizing for Generalization in Machine Learning with Cross-Validation Gradients
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that th…
BIG-bench Machine LearningHyperparameter OptimizationregressionGradient-based Hyperparameter Optimization through Reversible Learning
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives back…
Hyperparameter OptimizationScalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
Hyperparameter selection generally relies on running multiple full training trials, with selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during tr…
Hyperparameter OptimizationStability and Generalization of Bilevel Programming in Hyperparameter Optimization
The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties…
Hyperparameter Optimization