Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
With the rapid scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for improving deployment efficiency. This work aims to advance the compression beyond the weights to neuron activations. We propose the joint regularization technique which simultaneously regulates the distribution of weights and activations. By distinguishing and leveraging the significance difference among neuron responses and connections during learning, the jointly pruned network, namely \textit{JPnet}, optimizes the sparsity of activations and weights for improving execution efficiency. The derived deep sparsification of JPnet reveals more optimization space for the existing DNN accelerators dedicated for sparse matrix operations. We thoroughly evaluate the effectiveness of joint regularization through various network models with different activation functions and on different datasets. With $0.4\%$ degradation constraint on inference accuracy, a JPnet can save $72.3\% \sim 98.8\%$ of computation cost compared to the original dense models, with up to $5.2\times$ and $12.3\times$ reductions in activation and weight numbers, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Efficient Neural NetworkModel CompressionNetwork PruningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
Quantization and pruning are core techniques used to reduce the inference costs of deep neural networks. State-of-the-art quantization techniques are currently applied to both the weights and activations; however, prunin…
Network PruningQuantizationLearning Sparse Low-Precision Neural Networks With Learnable Regularization
We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backwa…
Image Super-ResolutionL2 RegularizationQuantizationSuper-ResolutionIntegral Pruning on Activations and Weights for Efficient Neural Networks
With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compr…
Model CompressionEnhancing the Regularization Effect of Weight Pruning in Artificial Neural Networks
Artificial neural networks (ANNs) may not be worth their computational/memory costs when used in mobile phones or embedded devices. Parameter-pruning algorithms combat these costs, with some algorithms capable of removin…
General Classificationimage-classificationImage ClassificationBinary Stochastic Filtering: feature selection and beyond
Feature selection is one of the most decisive tools in understanding data and machine learning models. Among other methods, sparsity induced by $L^{1}$ penalty is one of the simplest and best studied approaches to this p…
feature selection