MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
In neural architecture search (NAS), differentiable architecture search (DARTS) has recently attracted much attention due to its high efficiency. It defines an over-parameterized network with mixed edges, each of which represents all operator candidates, and jointly optimizes the weights of the network and its architecture in an alternating manner. However, this method finds a model with the weights converging faster than the others, and such a model with fastest convergence often leads to overfitting. Accordingly, the resulting model cannot always be well-generalized. To overcome this problem, we propose a method called minimum stable rank DARTS (MSR-DARTS), for finding a model with the best generalization error by replacing architecture optimization with the selection process using the minimum stable rank criterion. Specifically, a convolution operator is represented by a matrix, and MSR-DARTS selects the one with the smallest stable rank. We evaluated MSR-DARTS on CIFAR-10 and ImageNet datasets. It achieves an error rate of 2.54% with 4.0M parameters within 0.3 GPU-days on CIFAR-10, and a top-1 error rate of 23.9% on ImageNet. The official code is available at https://github.com/mtaecchhi/msrdarts.git.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUNeural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Single-DARTS: Towards Stable Architecture Search
Differentiable architecture search (DARTS) marks a milestone in Neural Architecture Search (NAS), boasting simplicity and small search costs. However, DARTS still suffers from frequent performance collapse, which happens…
Neural Architecture SearchiDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the search…
GPUNeural Architecture SearchNeighborhood-Aware Neural Architecture Search
Existing neural architecture search (NAS) methods often return an architecture with good search performance but generalizes poorly to the test setting. To achieve better generalization, we propose a novel neighborhood-aw…
Neural Architecture SearchDARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing and improving parts of the DARTS framew…
Bilevel OptimizationNeural Architecture SearchSchedulingDU-DARTS: Decreasing the Uncertainty of Differentiable Architecture Search
Differentiable Neural Architecture Search (DARTS) recently attracts a lot of research attention because of its high efficiency. However, the competition of candidate operations in DARTS introduces high uncertainty for se…
Neural Architecture Search