Long-Short Range Context Neural Networks for Language Modeling
The goal of language modeling techniques is to capture the statistical and structural properties of natural languages from training corpora. This task typically involves the learning of short range dependencies, which generally model the syntactic properties of a language and/or long range dependencies, which are semantic in nature. We propose in this paper a new multi-span architecture, which separately models the short and long context information while it dynamically merges them to perform the language modeling task. This is done through a novel recurrent Long-Short Range Context (LSRC) network, which explicitly models the local (short) and global (long) context using two separate hidden states that evolve in time. This new architecture is an adaptation of the Long-Short Term Memory network (LSTM) to take into account the linguistic properties. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art language modeling techniques.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingText CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation
Neural audio codecs, used as speech tokenizers, have demonstrated remarkable potential in the field of speech generation. However, to ensure high-fidelity audio reconstruction, neural audio codecs typically encode audio …
Language ModelingLanguage Modellingtext-to-speechText to SpeechMulti-cell LSTM Based Neural Language Model
Language models, being at the heart of many NLP problems, are always of great interest to researchers. Neural language models come with the advantage of distributed representations and long range contexts. With its parti…
Language ModelingLanguage ModellingmodelToken Weighting for Long-Range Language Modeling
Many applications of large language models (LLMs) require long-context understanding, but models continue to struggle with such tasks. We hypothesize that conventional next-token prediction training could contribute to t…
Language ModelingLanguage ModellingLong-Context UnderstandingRWKV-X: A Linear Complexity Hybrid Language Model
In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybr…
Language ModelingLanguage ModellingmodelLinear Attention via Orthogonal Memory
Efficient attentions have greatly improved the computational efficiency of Transformers. However, most existing linear attention mechanisms suffer from an \emph{efficiency degradation} problem, leading to inefficiencies …
Causal Language ModelingComputational EfficiencyLanguage ModelingLanguage Modelling