Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search
Developing effective surrogates (performance predictors) for Neural Architecture Search (NAS) typically requires expensive fine-tuning or the engineering of complex representations. We propose a low-cost embedding strategy that leverages the inductive bias of Language Models (LMs) to eliminate these overheads. By representing architectures as PyTorch class definition text, we demonstrate that off-the-shelf LMs act as competitive feature extractors without NAS-specialized fine-tuning. The final predictor is constructed by passing the extracted Code-Oriented LM Embeddings (COLE) through a lightweight regression head. We also investigate strategies to improve embedding quality and utilization. Our experiments on the NAS-Bench-201 and einspace search spaces reveal that raw code inputs yield higher predictive performance than other text-based encodings (e.g., ONNX-to-text encodings) when using frozen LMs. We also observe COLE drives superior surrogate-assisted search using the BANANAS algorithm in NAS-Bench-201. When optimizing for CIFAR-100 performance, replacing structural path encodings with COLE for architecture representation allows for a 34% decrease in the evaluation budget required to reach within 1% of the fittest architecture in the search space (by test accuracy). As any neural architecture can be represented as code, these findings establish COLE as a versatile and efficient foundation for advancing NAS.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
Surrogate-Assisted Evolution for Efficient Multi-branch Connection Design in Deep Neural Networks
State-of-the-art Deep Neural Networks (DNNs) often incorporate multi-branch connections, enabling multi-scale feature extraction and enhancing the capture of diverse features. This design improves network capacity and ge…
Evolutionary AlgorithmsMulti-surrogate Assisted Efficient Global Optimization for Discrete Problems
Decades of progress in simulation-based surrogate-assisted optimization and unprecedented growth in computational power have enabled researchers and practitioners to optimize previously intractable complex engineering pr…
global-optimizationManagementSurrogate-assisted Particle Swarm Optimisation for Evolving Variable-length Transferable Blocks for Image Classification
Deep convolutional neural networks have demonstrated promising performance on image classification tasks, but the manual design process becomes more and more complex due to the fast depth growth and the increasingly comp…
General ClassificationGPUimage-classificationImage Classification+1Adjoint-based shape optimization of a ship hull using a Conditional Variational Autoencoder (CVAE) assisted propulsion surrogate model
Adjoint-based shape optimization of ship hulls is a powerful tool for addressing high-dimensional design problems in naval architecture, particularly in minimizing the ship resistance. However, its application to vessels…
Exploring the effectiveness of surrogate-assisted evolutionary algorithms on the batch processing problem
Real-world optimisation problems typically have objective functions which cannot be expressed analytically. These optimisation problems are evaluated through expensive physical experiments or simulations. Cheap approxima…
Evolutionary Algorithms