Why Are Kronecker Products So Effective?
In this blog post we will review the layer scheme proposed by the authors of the paper "Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $$1/n$$ Parameters", which links quaternion-based Neural Networks with the Kronecker Product, and later explain how the Kronecker Product provides a connection between the paper and a parallel line of research into parameter efficient Neural Networks for Computer Vision based on Singular Value Decomposition.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Dominant Z-Eigenpairs of Tensor Kronecker Products are Decoupled and Applications to Higher-Order Graph Matching
Tensor Kronecker products, the natural generalization of the matrix Kronecker product, are independently emerging in multiple research communities. Like their matrix counterpart, the tensor generalization gives structure…
Graph MatchingKrony-PT: GPT2 compressed with Kronecker Products
We introduce Krony-PT, a compression technique of GPT2 \citep{radford2019language} based on Kronecker Products. We specifically target the MLP layers of each transformer layer, and systematically compress the feed forwar…
Language ModelingLanguage ModellingCompression of Fully-Connected Layer in Neural Network by Kronecker Product
In this paper we propose and study a technique to reduce the number of parameters and computation time in fully-connected layers of neural networks using Kronecker product, at a mild cost of the prediction quality. The t…
Hybrid Kronecker Product Decomposition and Approximation
Discovering the underlying low dimensional structure of high dimensional data has attracted a significant amount of researches recently and has shown to have a wide range of applications. As an effective dimension reduct…
Dimensionality ReductionExploiting Local Structures with the Kronecker Layer in Convolutional Networks
In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fu…
Scene Text Recognition