Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
Increasing the capacity of recurrent neural networks (RNN) usually involves augmenting the size of the hidden layer, with significant increase of computational cost. Recurrent neural tensor networks (RNTN) increase capacity using distinct hidden layer weights for each word, but with greater costs in memory usage. In this paper, we introduce restricted recurrent neural tensor networks (r-RNTN) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights for infrequent words. Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve language model performance over RNNs using only a small fraction of the parameters of unrestricted RNTNs. These results hold for r-RNTNs using Gated Recurrent Units and Long Short-Term Memory.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingTensor NetworksSimilar Papers 제목 키워드 기반
Enhancing Modern Supervised Word Sense Disambiguation Models by Semantic Lexical Resources
Supervised models for Word Sense Disambiguation (WSD) currently yield to state-of-the-art results in the most popular benchmarks. Despite the recent introduction of Word Embeddings and Recurrent Neural Networks to design…
Word EmbeddingsWord Sense DisambiguationNoise-Robust ASR for the third 'CHiME' Challenge Exploiting Time-Frequency Masking based Multi-Channel Speech Enhancement and Recurrent Neural Network
In this paper, the Lingban entry to the third 'CHiME' speech separation and recognition challenge is presented. A time-frequency masking based speech enhancement front-end is proposed to suppress the environmental noise …
Language ModelingLanguage ModellingSpeech Enhancementspeech-recognition+2Tensor Restricted Isometry Property Analysis For a Large Class of Random Measurement Ensembles
In previous work, theoretical analysis based on the tensor Restricted Isometry Property (t-RIP) established the robust recovery guarantees of a low-tubal-rank tensor. The obtained sufficient conditions depend strongly on…
Towards efficient keyword spotting using spike-based time difference encoders
Keyword spotting in edge devices is becoming increasingly important as voice-activated assistants are widely used. However, its deployment is often limited by the extreme low-power constraints of the target embedded syst…
Keyword SpottingLanguage Modeling Using Tensor Trains
We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by t…
Language ModelingLanguage Modelling