Understanding and Robustifying Differentiable Architecture Search
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.
Code (1)
Tasks
Disparity Estimationimage-classificationImage ClassificationLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation
Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the s…
DisentanglementNeural Architecture SearchThe devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol
Neural Architecture Search (NAS) has been widely adopted to design neural networks for various computer vision tasks. One of its most promising subdomains is differentiable NAS (DNAS), where the optimal architecture is f…
GPUNeural Architecture SearchOperation-Level Early Stopping for Robustifying Differentiable NAS
Differentiable NAS (DARTS) is a simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks. % Nevertheless, DARTS still encounters several robustness issue…
Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
Differentiable architecture search (DARTS) is a promising end to end NAS method which directly optimizes the architecture parameters through general gradient descent. However, DARTS is brittle to the catastrophic failure…
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
Despite the fast development of differentiable architecture search (DARTS), it suffers from long-standing performance instability, which extremely limits its application. Existing robustifying methods draw clues from the…
AutoMLNeural Architecture Search