A Deeper Look at Zero-Cost Proxies for Lightweight NAS
While early algorithms for neural architecture search (NAS) used extreme computational budgets, recent techniques have aimed to lower the computation time. Very recently, a new family of techniques were introduced to approximate the performance of neural architectures in just five seconds: so-called "zero-cost proxies" (Mellor et al. 2020; Abdelfattah et al. 2021). However, two recent papers have showed that simple baselines such as "number of parameters" are competitive, casting doubt on the efficacy of zero-cost proxies (Ning et al. 2021, Chen et al. 2021). In this blog post, we take a deeper look at zero-cost proxies for NAS. We survey prior work and then run new experiments using the recent NAS-Bench-360 and TransNAS-Bench-101 benchmarks, which give a much more diverse set of datasets and tasks than all prior work. We find that there are no clear "best" zero-cost proxies across all tasks, and we confirm that simple baselines are consistently competitive with proposed zero-cost proxies. However, we also conclude that zero-cost proxies have great potential in NAS, especially when used in combination with other methods or to improve the performance of existing methods at very little additional cost. Overall, we provide a landscape overview of this promising area, highlight strengths and weaknesses, and shed light on future research in this direction.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
Zero-Cost Proxies for Lightweight NAS
Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the …
Neural Architecture SearchParZC: Parametric Zero-Cost Proxies for Efficient NAS
Recent advancements in Zero-shot Neural Architecture Search (NAS) highlight the efficacy of zero-cost proxies in various NAS benchmarks. Several studies propose the automated design of zero-cost proxies to achieve SOTA p…
Neural Architecture SearchEZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring
Neural Architecture Search (NAS) has significantly improved productivity in the design and deployment of neural networks (NN). As NAS typically evaluates multiple models by training them partially or completely, the impr…
Neural Architecture SearchAn Evaluation of Zero-Cost Proxies -- from Neural Architecture Performance to Model Robustness
Zero-cost proxies are nowadays frequently studied and used to search for neural architectures. They show an impressive ability to predict the performance of architectures by making use of their untrained weights. These t…
Feature ImportanceProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
Designing neural architectures requires immense manual efforts. This has promoted the development of neural architecture search (NAS) to automate the design. While previous NAS methods achieve promising results but run s…
Bayesian OptimizationNeural Architecture Search