Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fully-connected layers, by replacing the large weight matrices by combinations of multiple Kronecker products of smaller matrices. Just as the Kronecker product is a generalization of the outer product from vectors to matrices, our method is a generalization of the low rank approximation method for convolution neural networks. We also introduce combinations of different shapes of Kronecker product to increase modeling capacity. Experiments on SVHN, scene text recognition and ImageNet dataset demonstrate that we can achieve $3.3 \times$ speedup or $3.6 \times$ parameter reduction with less than 1\% drop in accuracy, showing the effectiveness and efficiency of our method. Moreover, the computation efficiency of Kronecker layer makes using larger feature map possible, which in turn enables us to outperform the previous state-of-the-art on both SVHN(digit recognition) and CASIA-HWDB (handwritten Chinese character recognition) datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Scene Text RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition
Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge devices. In this paper, we reduce memory…
image-classificationImage ClassificationKnowledge DistillationNeural Network CompressionSKFAC:Training Neural Networks with Faster Kronecker-Factored Approximate Curvature
The bottleneck of computation burden limits the widespread use of the 2nd order optimization algorithms for training deep neural networks. In this paper, we present a computationally efficient approximation for natural g…
Dimensionality ReductionSKFAC: Training Neural Networks With Faster Kronecker-Factored Approximate Curvature
The bottleneck of computation burden limits the widespread use of the 2nd order optimization algorithms for training deep neural networks. In this paper, we present a computationally efficient approximation for natur…
Dimensionality ReductionKronecker-factored Quasi-Newton Methods for Deep Learning
Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training par…
Deep LearningSecond-order methodsEfficient Asynchronize Stochastic Gradient Algorithm with Structured Data
Deep learning has achieved impressive success in a variety of fields because of its good generalization. However, it has been a challenging problem to quickly train a neural network with a large number of layers. The exi…