Bilevel Optimization for Neural Architecture Search
Bilevel optimization has become an influential and widely adopted framework for addressing hierarchical optimization problems in machine learning, providing an effective approach to modeling the interaction between two levels of optimization, with applications such as hyperparameter tuning, meta-learning, adversarial training, and data poisoning. Neural Architecture Search (NAS), a subfield of hyperparameter optimization, is a prime example of a bilevel optimization problem, with architecture parameters optimized at the outer-level and network weights optimized at the inner level. This paper presents a structured overview of NAS through the lens of bilevel optimization. We categorize existing NAS approaches into two main classes: sampling-based methods, which search optimal architectures using different architecture samplers, and bilevel theory-based methods, which solve the architecture search problem using bilevel optimization principles. We further highlight our current research direction, wherein the bilevel NAS formulation is addressed through an auxiliary mathematical programming framework. This framework enables the systematic integration of second-order information from the model's training loss function and ensures the optimality of the model parameters while modifying architecture parameters. By simultaneously updating the architecture and model parameters along their respective optimal descent directions derived from the auxiliary mathematical program, these methods achieve more principled and theoretically consistent results. The same auxiliary program can also be used for simultaneous hyperparameter and model fine-tuning. A comparative analysis shows that bilevel theory-based approaches generally outperform sampling-based methods, both in accuracy and efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Hyperparameter OptimizationNeural Architecture SearchBilevel OptimizationSimilar Papers 제목 키워드 기반
A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima
Bilevel optimization has found extensive applications in modern machine learning problems such as hyperparameter optimization, neural architecture search, meta-learning, etc. While bilevel problems with a unique inner mi…
Bilevel OptimizationHyperparameter OptimizationMeta-LearningNeural Architecture SearchOn Weight-Sharing and Bilevel Optimization in Architecture Search
Weight-sharing—the simultaneous optimization of multiple neural networks using the same parameters—has emerged as a key component of state-of-the-art neural architecture search. However, its success is poorly understood …
Bilevel Optimizationfeature selectionNeural Architecture SearchBilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
Bilevel optimization has become a powerful framework in various machine learning applications including meta-learning, hyperparameter optimization, and network architecture search. There are generally two classes of bile…
BIG-bench Machine LearningBilevel OptimizationHyperparameter OptimizationMeta-LearningA Globally Convergent Gradient-based Bilevel Hyperparameter Optimization Method
Hyperparameter optimization in machine learning is often achieved using naive techniques that only lead to an approximate set of hyperparameters. Although techniques such as Bayesian optimization perform an intelligent s…
Bayesian OptimizationBilevel OptimizationHyperparameter OptimizationInexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems
Two-level stochastic optimization formulations have become instrumental in a number of machine learning contexts such as continual learning, neural architecture search, adversarial learning, and hyperparameter tuning. Pr…
BIG-bench Machine LearningBilevel OptimizationContinual LearningNeural Architecture Search+1