A Tensorized Transformer for Language Modeling
Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.
Code (1)
Tasks
DecoderLanguage ModelingLanguage ModellingMachine TranslationTensor DecompositionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Tensorized Transformer for Dynamical Systems Modeling
The identification of nonlinear dynamics from observations is essential for the alignment of the theoretical ideas and experimental data. The last, in turn, is often corrupted by the side effects and noise of different n…
Language ModelingLanguage ModellingLong Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning
As the demand for processing extended textual data grows, the ability to handle long-range dependencies and maintain computational efficiency is more critical than ever. One of the key issues for long-sequence modeling u…
Computational EfficiencyTensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram att…
A tensorized logic programming language for large-scale data
We introduce a new logic programming language T-PRISM based on tensor embeddings. Our embedding scheme is a modification of the distribution semantics in PRISM, one of the state-of-the-art probabilistic logic programming…
Knowledge GraphsPartial Tensorized Transformers for Natural Language Processing
The transformer architecture has revolutionized Natural Language Processing (NLP) and other machine-learning tasks, due to its unprecedented accuracy. However, their extensive memory and parameter requirements often hind…
Tensor Decomposition