A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search

2022-04-08 · Stephen Cha, Taehyeon Kim, Hayeon Lee, Se-Young Yun

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.

📄 PDF Abstract BibTeX arXiv:2204.03916

Code (0)

등록된 구현이 없습니다.

Tasks

Domain Generalization Image Classification Neural Architecture Search object-detection Object Detection Semantic SegmentationSurvey

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Position-Wise Feed-Forward Layer 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

A Survey on Spatio-temporal Data Analytics Systems

2021-03-17 · Md Mahbub Alam, Luis Torgo, Albert Bifet

Due to the surge of spatio-temporal data volume, the popularity of location-based services and applications, and the importance of extracted knowledge from spatio-temporal data to solve a wide range of real-world problem…

Survey

Dynamic Neural Networks: A Survey

2021-02-09 · Yizeng Han, Gao Huang, Shiji Song, Le Yang 외

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or…

Computational Efficiency Decision Making Dynamic neural networks Survey

Survey of Federated Learning Models for Spatial-Temporal Mobility Applications

2023-05-09 · Yacine Belal, Sonia Ben Mokhtar, Hamed Haddadi, Jaron Wang 외

Federated learning involves training statistical models over edge devices such as mobile phones such that the training data is kept local. Federated Learning (FL) can serve as an ideal candidate for training spatial temp…

Community Detection Federated Learning Recommendation Systems Traffic Prediction

Natural Language Interfaces for Spatial and Temporal Databases: A Comprehensive Overview of Methods, Taxonomy, and Future Directions

2026-03-24 · Samya Acharja, Kanchan Chowdhury arxiv

The task of building a natural language interface to a database, known as NLIDB, has recently gained significant attention from both the database and Natural Language Processing (NLP) communities. With the proliferation …

Rethinking Architecture Selection in Differentiable NAS

2021-08-10 · ICLR 2021 1 · Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang 외

Differentiable Neural Architecture Search is one of the most popular Neural Architecture Search (NAS) methods for its search efficiency and simplicity, accomplished by jointly optimizing the model weight and architecture…

Neural Architecture Search