Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models
In this paper, we propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) (Zhang et al., 2015b) in neural languages modelling, called dual-FOFE. The main idea of dual-FOFE is that it allows to use two different forgetting factors so that it can avoid the trade-off in choosing either a small or large values for the single forgetting factor. In our experiments, we have compared the dual-FOFE based neural network language models (NNLM) against the original FOFE counterparts and various traditional NNLMs. Our results on the challenging Google Billion word corpus show that both FOFE and dual FOFE yield very strong performance while significantly reducing the computational complexity over other NNLMs. Furthermore, the proposed dual-FOFE method further gives over 10{\%} improvement in perplexity over the original FOFE model.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationSpeech RecognitionText SummarizationSimilar Papers 제목 키워드 기반
The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models
A Fixed-Size Encoding Method for Variable-Length Sequences with its Application to Neural Network Language Models
In this paper, we propose the new fixed-size ordinally-forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence of words into a fixed-size representation. FOFE can model the word o…
Word Embeddings based on Fixed-Size Ordinally Forgetting Encoding
In this paper, we propose to learn word embeddings based on the recent fixed-size ordinally forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence into a fixed-size representatio…
Language ModelingLanguage ModellingSemantic Textual SimilarityWord Embeddings+1Fixed-Size Ordinally Forgetting Encoding Based Word Sense Disambiguation
In this paper, we present our method of using fixed-size ordinally forgetting encoding (FOFE) to solve the word sense disambiguation (WSD) problem. FOFE enables us to encode variable-length sequence of words into a theor…
Language ModelingLanguage ModellingWord Sense DisambiguationDual-FOFE-net Neural Models for Entity Linking with PageRank
This paper presents a simple and computationally efficient approach for entity linking (EL), compared with recurrent neural networks (RNNs) or convolutional neural networks (CNNs), by making use of feedforward neural net…
ClusteringEntity LinkingSentence