Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization
Neural architecture search (NAS) and network pruning are widely studied efficient AI techniques, but not yet perfect. NAS performs exhaustive candidate architecture search, incurring tremendous search cost. Though (structured) pruning can simply shrink model dimension, it remains unclear how to decide the per-layer sparsity automatically and optimally. In this work, we revisit the problem of layer-width optimization and propose Pruning-as-Search (PaS), an end-to-end channel pruning method to search out desired sub-network automatically and efficiently. Specifically, we add a depth-wise binary convolution to learn pruning policies directly through gradient descent. By combining the structural reparameterization and PaS, we successfully searched out a new family of VGG-like and lightweight networks, which enable the flexibility of arbitrary width with respect to each layer instead of each stage. Experimental results show that our proposed architecture outperforms prior arts by around $1.0\%$ top-1 accuracy under similar inference speed on ImageNet-1000 classification task. Furthermore, we demonstrate the effectiveness of our width search on complex tasks including instance segmentation and image translation. Code and models are released.
Code (1)
Tasks
Instance SegmentationNetwork PruningNeural Architecture SearchSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
The convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead against efficient deployment. Structured (channel) pruning is usually applied to reduce t…
Neural Architecture SearchNetwork Pruning via Transformable Architecture Search
Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters…
Knowledge DistillationNetwork PruningNeural Architecture SearchTransfer LearningRevisiting Random Channel Pruning for Neural Network Compression
Channel (or 3D filter) pruning serves as an effective way to accelerate the inference of neural networks. There has been a flurry of algorithms that try to solve this practical problem, each being claimed effective in so…
Neural Network CompressionAdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance
Channel pruning is one of the major compression approaches for deep neural networks. While previous pruning methods have mostly focused on identifying unimportant channels, channel pruning is considered as a special case…
image-classificationImage ClassificationNeural Architecture SearchMetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for an…
AutoMLMeta-Learning