Tensorized Embedding Layers for Efficient Model Compression
The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.
Code (1)
Tasks
Language ModellingMachine TranslationmodelModel CompressionSentiment AnalysisSimilar Papers 제목 키워드 기반
Tensorized Embedding Layers
The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices…
MorphTE: Injecting Morphology in Tensorized Embeddings
In the era of deep learning, word embeddings are essential when dealing with text tasks. However, storing and accessing these embeddings requires a large amount of space. This is not conducive to the deployment of these …
Learning Word EmbeddingsMachine TranslationQuestion AnsweringTranslation+1Application of Tensorized Neural Networks for Cloud Classification
Convolutional neural networks (CNNs) have gained widespread usage across various fields such as weather forecasting, computer vision, autonomous driving, and medical image analysis due to its exceptional ability to extra…
Autonomous DrivingClassificationData CompressionMedical Image Analysis+2Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram att…
Bayesian Tensorized Neural Networks with Automatic Rank Selection
Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the…
Model CompressionTensor Decomposition