NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

2022-10-06 · Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, Frank Hutter

Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS). Recent work has shown that these techniques show great promise, but certain aspects, such as evaluating and exploiting their complementary strengths, are under-studied. In this work, we create NAS-Bench-Suite: we evaluate 13 ZC proxies across 28 tasks, creating by far the largest dataset (and unified codebase) for ZC proxies, enabling orders-of-magnitude faster experiments on ZC proxies, while avoiding confounding factors stemming from different implementations. To demonstrate the usefulness of NAS-Bench-Suite, we run a large-scale analysis of ZC proxies, including a bias analysis, and the first information-theoretic analysis which concludes that ZC proxies capture substantial complementary information. Motivated by these findings, we present a procedure to improve the performance of ZC proxies by reducing biases such as cell size, and we also show that incorporating all 13 ZC proxies into the surrogate models used by NAS algorithms can improve their predictive performance by up to 42%. Our code and datasets are available at https://github.com/automl/naslib/tree/zerocost.

📄 PDF Abstract BibTeX arXiv:2210.03230

Code (1)

automl/NASLib 공식 구현 pytorch

Tasks

AutoML Neural Architecture Search

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

RITA: a Study on Scaling Up Generative Protein Sequence Models

2022-05-11 · Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli 외

In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Su…

Prediction Protein Design

OwkinZero: Accelerating Biological Discovery with AI

2025-08-22 · Nathan Bigaud, Vincent Cabeli, Meltem Gürel, Arthur Pignet 외 arxiv

While large language models (LLMs) are rapidly advancing scientific research, they continue to struggle with core biological reasoning tasks essential for translational and biomedical discovery. To address this limitatio…

Reinforcement Learning Drug Discovery

NoveltyRank: A Retrieval-Augmented Framework for Conceptual Novelty Estimation in AI Research

2025-12-12 · Zhengxu Yan, Han Li, Yuming Feng arxiv

The accelerating pace of scientific publication makes it difficult to identify truly original research among incremental work. We propose a framework for estimating the conceptual novelty of research papers by combining …

Representation Learning Binary Classification

ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition

2025-04-17 · Haidar Khan, Hisham A. Alyahya, Yazeed Alnumay, M Saiful Bari 외

Evaluating the capabilities of Large Language Models (LLMs) has traditionally relied on static benchmark datasets, human assessments, or model-based evaluations - methods that often suffer from overfitting, high costs, a…

model

From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents

2026-04-02 · Nikolai Ludwig, Wasi Uddin Ahmad, Somshubra Majumdar, Boris Ginsburg arxiv

We introduce SWE-ZERO to SWE-HERO, a two-stage SFT recipe that achieves state-of-the-art results on SWE-bench by distilling open-weight frontier LLMs. Our pipeline replaces resource-heavy dependencies with an evolutionar…