A Factorized Recurrent Neural Network based architecture for medium to large vocabulary Language Modelling
Statistical language models are central to many applications that use semantics. Recurrent Neural Networks (RNN) are known to produce state of the art results for language modelling, outperforming their traditional n-gram counterparts in many cases. To generate a probability distribution across a vocabulary, these models require a softmax output layer that linearly increases in size with the size of the vocabulary. Large vocabularies need a commensurately large softmax layer and training them on typical laptops/PCs requires significant time and machine resources. In this paper we present a new technique for implementing RNN based large vocabulary language models that substantially speeds up computation while optimally using the limited memory resources. Our technique, while building on the notion of factorizing the output layer by having multiple output layers, improves on the earlier work by substantially optimizing on the individual output layer size and also eliminating the need for a multistep prediction process.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LongFNT: Long-form Speech Recognition with Factorized Neural Transducer
Traditional automatic speech recognition~(ASR) systems usually focus on individual utterances, without considering long-form speech with useful historical information, which is more practical in real scenarios. Simply at…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)FormLanguage Modeling+4Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models
Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) mo…
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionLong Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNN…
Handwriting RecognitionLanguage ModelingLanguage Modellingspeech-recognition+1Factorized Neural Transducer for Efficient Language Model Adaptation
In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance. Neural Transducer based models are increasingly popular in…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+3Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
In order to integrate uncertainty estimates into deep time-series modelling, Kalman Filters (KFs) (Kalman et al., 1960) have been integrated with deep learning models, however, such approaches typically rely on approxima…
Image ImputationImputationTime SeriesTime Series Analysis+2