NAS-Bench-101: Towards Reproducible Neural Architecture Search
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
Code (4)
Tasks
BenchmarkingNeural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search
Graph neural architecture search (GraphNAS) has recently aroused considerable attention in both academia and industry. However, two key challenges seriously hinder the further research of GraphNAS. First, since there is …
BenchmarkingGraph Neural NetworkNeural Architecture SearchDifferentiable Neural Architecture Transformation for Reproducible Architecture Improvement
Recently, Neural Architecture Search (NAS) methods are introduced and show impressive performance on many benchmarks. Among those NAS studies, Neural Architecture Transformer (NAT) aims to improve the given neural archit…
Neural Architecture SearchNAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition
Powered by innovations in novel architecture design, noise tolerance techniques and increasing model capacity, Automatic Speech Recognition (ASR) made giant strides in improving prediction accuracies over the past decade…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Neural Architecture Searchspeech-recognition+1RAISE: RAG Design as an Architecture Search Problem
Retrieval-augmented generation (RAG) systems expose numerous design choices spanning query rewriting, chunking, retrieval depth, reranking, and context compression. In practice, these choices are often configured through…
Hyperparameter OptimizationEfficient Adaptive Transformer: An Empirical Study and Reproducible Framework
The Efficient Adaptive Transformer (EAT) framework unifies three adaptive efficiency techniques - progressive token pruning, sparse attention, and dynamic early exiting - into a single, reproducible architecture for inpu…