Feedforward Sequential Memory Neural Networks without Recurrent Feedback
We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback. The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers. In this work, we have applied FSMN to several language modeling (LM) tasks. Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history. Experiments have shown that FSMN based language models can significantly outperform not only feedforward neural network (FNN) based LMs but also the popular recurrent neural network (RNN) LMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency
In this paper, we propose a novel neural network structure, namely \emph{feedforward sequential memory networks (FSMN)}, to model long-term dependency in time series without using recurrent feedback. The proposed FSMN is…
Language Modellingspeech-recognitionSpeech RecognitionTime Series+1A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation
We describe recurrent neural networks (RNNs), which have attracted great attention on sequential tasks, such as handwriting recognition, speech recognition and image to text. However, compared to general feedforward neur…
Handwriting RecognitionImage to textspeech-recognitionSpeech RecognitionAddressing Some Limitations of Transformers with Feedback Memory
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while proces…
Language ModelingLanguage ModellingMachine TranslationReinforcement Learning+1Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks
The abundant recurrent horizontal and feedback connections in the primate visual cortex are thought to play an important role in bringing global and semantic contextual information to early visual areas during perceptual…
Surprisal-Driven Feedback in Recurrent Networks
Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missi…