Compression of Fully-Connected Layer in Neural Network by Kronecker Product
In this paper we propose and study a technique to reduce the number of parameters and computation time in fully-connected layers of neural networks using Kronecker product, at a mild cost of the prediction quality. The technique proceeds by replacing Fully-Connected layers with so-called Kronecker Fully-Connected layers, where the weight matrices of the FC layers are approximated by linear combinations of multiple Kronecker products of smaller matrices. In particular, given a model trained on SVHN dataset, we are able to construct a new KFC model with 73\% reduction in total number of parameters, while the error only rises mildly. In contrast, using low-rank method can only achieve 35\% reduction in total number of parameters given similar quality degradation allowance. If we only compare the KFC layer with its counterpart fully-connected layer, the reduction in the number of parameters exceeds 99\%. The amount of computation is also reduced as we replace matrix product of the large matrices in FC layers with matrix products of a few smaller matrices in KFC layers. Further experiments on MNIST, SVHN and some Chinese Character recognition models also demonstrate effectiveness of our technique.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Why Are Kronecker Products So Effective?
In this blog post we will review the layer scheme proposed by the authors of the paper "Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $$1/n$$ Parameters", which lin…
Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fu…
Scene Text RecognitionConvolutional Neural Network Compression through Generalized Kronecker Product Decomposition
Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge devices. In this paper, we reduce memory…
image-classificationImage ClassificationKnowledge DistillationNeural Network CompressionWeight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures
Deep learning using neural networks is an effective technique for generating models of complex data. However, training such models can be expensive when networks have large model capacity resulting from a large number of…
Deep LearningDimensionality ReductionCompressing Language Models using Doped Kronecker Products
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language P…
Language ModelingLanguage ModellingLarge Language Model