Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights
Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) aims to bridge this gap by following a well-defined optimization paradigm which systematically looks for the best architecture, given objective criterion such as maximal classification accuracy. The main limitation of NAS is however its astronomical computational cost, as it typically requires training each candidate network architecture from scratch.In this paper, we aim to alleviate this limitation by proposing a novel training-free proxy for image classification accuracy based on Fisher Information. The proposed proxy has a strong theoretical background in statistics and it allows estimating expected image classification accuracy of a given deep network without training the network, thus significantly reducing computational cost of standard NAS algorithms. Our training-free proxy achieves state-of-the-art results on three public datasets and in two search spaces, both when evaluated using previously proposed metrics, as well as using a new metric that we propose which we demonstrate is more informative for practical NAS applications. The source code is publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationNeural Architecture SearchSimilar Papers 제목 키워드 기반
Training-free Neural Architecture Search for RNNs and Transformers
Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures, offering an alternative to the laborious process of manually designing complex architectures. How…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights
Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) ai…
image-classificationImage ClassificationNeural Architecture SearchFree Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs
The covariance for clean data given a noisy observation is an important quantity in many conditional generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard dif…
Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
The recently proposed training-free NAS methods abandon the training phase and design various zero-cost proxies as scores to identify excellent architectures, arousing extreme computational efficiency for neural architec…
Computational EfficiencyNetwork PruningNeural Architecture SearchSensitivityInTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search
Training-free neural architecture search promises efficient discovery of high-performance networks without costly training. However, existing zero-cost proxies rely on fragmented heuristics that fail to capture the funda…
Neural Architecture Search