Understanding Architectures Learnt by Cell-based Neural Architecture Search
Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectures. In this paper, we first reveal that existing NAS algorithms (e.g., DARTS, ENAS) tend to favor architectures with wide and shallow cell structures. These favorable architectures consistently achieve fast convergence and are consequently selected by NAS algorithms. Our empirical and theoretical study further confirms that their fast convergence derives from their smooth loss landscape and accurate gradient information. Nonetheless, these architectures may not necessarily lead to better generalization performance compared with other candidate architectures in the same search space, and therefore further improvement is possible by revising existing NAS algorithms.
Code (1)
Tasks
image-classificationImage ClassificationLanguage ModelingLanguage ModellingNeural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Single-cell Subcellular Protein Localisation Using Novel Ensembles of Diverse Deep Architectures
Unravelling protein distributions within individual cells is key to understanding their function and state and indispensable to developing new treatments. Here we present the Hybrid subCellular Protein Localiser (HCPL), …
BIG-bench Machine LearningScheduled Differentiable Architecture Search for Visual Recognition
Convolutional Neural Networks (CNN) have been regarded as a capable class of models for visual recognition problems. Nevertheless, it is not trivial to develop generic and powerful network architectures, which requires s…
Video RecognitionOn the Privacy Risks of Cell-Based NAS Architectures
Existing studies on neural architecture search (NAS) mainly focus on efficiently and effectively searching for network architectures with better performance. Little progress has been made to systematically understand if …
Neural Architecture SearchLearning Architectures from an Extended Search Space for Language Modeling
Neural architecture search (NAS) has advanced significantly in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. In this paper, we extend the search space o…
ChunkingLanguage ModelingLanguage Modellingnamed-entity-recognition+4On Redundancy and Diversity in Cell-based Neural Architecture Search
Searching for the architecture cells is a dominant paradigm in NAS. However, little attention has been devoted to the analysis of the cell-based search spaces even though it is highly important for the continual developm…
DiversityNeural Architecture Search