Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
Neural architecture search (NAS) has fostered various fields of machine learning. Despite its prominent dedications, many have criticized the intrinsic limitations of high computational cost. We aim to ameliorate this by proposing a pretraining scheme that can be generally applied to controller-based NAS. Our method, locality-based self-supervised classification task, leverages the structural similarity of network architectures to obtain good architecture representations. We incorporate our method into neural architecture optimization (NAO) to analyze the pretrained embeddings and its effectiveness and highlight that adding metric learning loss brings a favorable impact on NAS. Our code is available at \url{https://github.com/Multi-Objective-NAS/self-supervised-nas}.
Code (1)
Tasks
Metric LearningNeural Architecture SearchSelf-Supervised LearningSimilar Papers 제목 키워드 기반
BERT-LSH: Reducing Absolute Compute For Attention
This study introduces a novel BERT-LSH model that incorporates Locality Sensitive Hashing (LSH) to approximate the attention mechanism in the BERT architecture. We examine the computational efficiency and performance of …
Computational EfficiencyConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transfo…
image-classificationImage ClassificationInductive BiasA Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy
Solutions to vision tasks in gastrointestinal endoscopy (GIE) conventionally use image encoders pretrained in a supervised manner with ImageNet-1k as backbones. However, the use of modern self-supervised pretraining algo…
Depth EstimationLandmark RecognitionMonocular Depth EstimationDoes GNN Pretraining Help Molecular Representation?
Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-super…
Drug Discoverymolecular representationCADS: Core-Aware Dynamic Scheduler for Multicore Memory Controllers
Memory controller scheduling is crucial in multicore processors, where DRAM bandwidth is shared. Since increased number of requests from multiple cores of processors becomes a source of bottleneck, scheduling the request…
FairnessReinforcement LearningReinforcement Learning (RL)Scheduling