Language Modeling Using Tensor Trains
We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by the tensor product of words, but computing the probabilities of sentences in a low-dimensional fashion. We demonstrate that the architectures of Second-order RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs are, essentially, special cases of TTLM. Experimental evaluations on real language modeling tasks show that the proposed variants of TTLM (i.e., TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks (RNNs) with low-scale of hidden units. (The code is available at https://github.com/shuishen112/tensortrainlm.)
Code (1)
Tasks
Language ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Tensor train decompositions on recurrent networks
Recurrent neural networks (RNN) such as long-short-term memory (LSTM) networks are essential in a multitude of daily live tasks such as speech, language, video, and multimodal learning. The shift from cloud to edge compu…
Learning parameter dependence for Fourier-based option pricing with tensor trains
A long-standing issue in mathematical finance is the speed-up of option pricing, especially for multi-asset options. A recent study has proposed to use tensor train learning algorithms to speed up Fourier transform (FT)-…
Tensor NetworksFast and Accurate Tensor Completion with Total Variation Regularized Tensor Trains
We propose a new tensor completion method based on tensor trains. The to-be-completed tensor is modeled as a low-rank tensor train, where we use the known tensor entries and their coordinates to update the tensor train. …
Video InpaintingDiffATS: Diffusion in Aligned Tensor Space
Direct diffusion modeling of high-resolution spatiotemporal fields is computationally challenging. Parameter-efficient primitives address this by representing high-dimensional data with a compact set of parameters. In th…
TTML: tensor trains for general supervised machine learning
This work proposes a novel general-purpose estimator for supervised machine learning (ML) based on tensor trains (TT). The estimator uses TTs to parametrize discretized functions, which are then optimized using Riemannia…
BIG-bench Machine Learning