DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing and improving parts of the DARTS framework have recently been proposed. By considering the problem as a constrained bilevel optimization, we present and analyze DARTS-PRIME, a variant including improvements to architectural weight update scheduling and regularization towards discretization. We propose a dynamic schedule based on per-minibatch network information to make architecture updates more informed, as well as proximity regularization to promote well-separated discretization. Our results in multiple domains show that DARTS-PRIME improves both performance and reliability, comparable to state-of-the-art in differentiable NAS.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilevel OptimizationNeural Architecture SearchSchedulingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and large…
Adversarial AttackNeural Architecture Search$Λ$-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells
Differentiable neural architecture search (DARTS) is a popular method for neural architecture search (NAS), which performs cell-search and utilizes continuous relaxation to improve the search efficiency via gradient-base…
Neural Architecture SearchHeterogeneous Learning Rate Scheduling for Neural Architecture Search on Long-Tailed Datasets
In this paper, we attempt to address the challenge of applying Neural Architecture Search (NAS) algorithms, specifically the Differentiable Architecture Search (DARTS), to long-tailed datasets where class distribution is…
Neural Architecture SearchScheduling$β$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search
Neural Architecture Search has attracted increasing attention in recent years. Among them, differential NAS approaches such as DARTS, have gained popularity for the search efficiency. However, they still suffer from thre…
Neural Architecture SearchsharpDARTS: Faster and More Accurate Differentiable Architecture Search
Neural Architecture Search (NAS) has been a source of dramatic improvements in neural network design, with recent results meeting or exceeding the performance of hand-tuned architectures. However, our understanding of ho…
Hyperparameter OptimizationImage ClassificationNeural Architecture Search