Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition
Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge devices. In this paper, we reduce memory usage and floating-point operations required by convolutional layers in CNNs. We compress these layers by generalizing the Kronecker Product Decomposition to apply to multidimensional tensors, leading to the Generalized Kronecker Product Decomposition (GKPD). Our approach yields a plug-and-play module that can be used as a drop-in replacement for any convolutional layer. Experimental results for image classification on CIFAR-10 and ImageNet datasets using ResNet, MobileNetv2 and SeNet architectures substantiate the effectiveness of our proposed approach. We find that GKPD outperforms state-of-the-art decomposition methods including Tensor-Train and Tensor-Ring as well as other relevant compression methods such as pruning and knowledge distillation.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationKnowledge DistillationNeural Network CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Compressing Language Models using Doped Kronecker Products
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language P…
Language ModelingLanguage ModellingLarge Language ModelExploiting Local Structures with the Kronecker Layer in Convolutional Networks
In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fu…
Scene Text RecognitionGeneralized vec trick for fast learning of pairwise kernel models
Pairwise learning corresponds to the supervised learning setting where the goal is to make predictions for pairs of objects. Prominent applications include predicting drug-target or protein-protein interactions, or custo…
Metric LearningFast Kronecker product kernel methods via generalized vec trick
Kronecker product kernel provides the standard approach in the kernel methods literature for learning from graph data, where edges are labeled and both start and end vertices have their own feature representations. The m…
Collaborative FilteringInformation RetrievalRetrievalCompressing RNNs for IoT devices by 15-38x using Kronecker Products
Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively…