Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
Recurrent neural network (RNN) based character-level language models (CLMs) are extremely useful for modeling out-of-vocabulary words by nature. However, their performance is generally much worse than the word-level language models (WLMs), since CLMs need to consider longer history of tokens to properly predict the next one. We address this problem by proposing hierarchical RNN architectures, which consist of multiple modules with different timescales. Despite the multi-timescale structures, the input and output layers operate with the character-level clock, which allows the existing RNN CLM training approaches to be directly applicable without any modifications. Our CLM models show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word Benchmark with only 2% of parameters. Also, we present real-time character-level end-to-end speech recognition examples on the Wall Street Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the proposed models results in better recognition accuracies even though the number of parameters are reduced to 30%.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network
Truecasing is the task of restoring the correct case (uppercase or lowercase) of noisy text generated either by an automatic system for speech recognition or machine translation or by humans. It improves the performance …
Language ModelingLanguage ModellingMachine Translationnamed-entity-recognition+7Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model
Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text. We propose a fast, accurate and compact two-level hierarchical word-and-character-based recurren…
Federated LearningLanguage ModelingLanguage ModellingGeneration of Hip-Hop Lyrics with Hierarchical Modeling and Conditional Templates
This paper addresses Hip-Hop lyric generation with conditional Neural Language Models. We develop a simple yet effective mechanism to extract and apply conditional templates from text snippets, and show{---}on the basis …
Language ModelingLanguage ModellingRhythmText GenerationHierarchical Multiscale Recurrent Neural Networks
Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolv…
Language ModellingThe Importance of Being Recurrent for Modeling Hierarchical Structure
Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks such as language modeling (Linzen et …
Language ModelingLanguage ModellingMachine TranslationTranslation