Transferrable Surrogates in Expressive Neural Architecture Search Spaces
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search
Neural Architecture Search (NAS) achieved many breakthroughs in recent years. In spite of its remarkable progress, many algorithms are restricted to particular search spaces. They also lack efficient mechanisms to reuse …
Meta-LearningMeta Reinforcement LearningNeural Architecture Searchreinforcement-learning+2TransNAS-Bench-101: Improving Transferrability and Generalizability of Cross-Task Neural Architecture Search
Recent breakthroughs of Neural Architecture Search (NAS) are extending the field's research scope towards a broader range of vision tasks and more diversified search spaces. While existing NAS methods mostly design archi…
GPUNeural Architecture SearchTransfer LearningConstruction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search …
AutoMLBayesian OptimizationNeural Architecture SearchNeural and spectral operator surrogates: unified construction and expression rate bounds
Approximation rates are analyzed for deep surrogates of maps between infinite-dimensional function spaces, arising e.g. as data-to-solution maps of linear and nonlinear partial differential equations. Specifically, we st…
On the Expressive Power of GNNs to Solve Linear SDPs
Semidefinite programs (SDPs) are a powerful framework for convex optimization and for constructing strong relaxations of hard combinatorial problems. However, solving large SDPs can be computationally expensive, motivati…