Scaling Recurrent Neural Network Language Models
This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, computational costs and memory. Our analysis shows that despite being more costly to train, RNNLMs obtain much lower perplexities on standard benchmarks than n-gram models. We train the largest known RNNs and present relative word error rates gains of 18% on an ASR task. We also present the new lowest perplexities on the recently released billion word language modelling benchmark, 1 BLEU point gain on machine translation and a 17% relative hit rate gain in word prediction.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Evaluating Computational Language Models with Scaling Properties of Natural Language
In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is c…
Text GenerationA Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation
Time perception is critical in our daily life. An important feature of time perception is temporal scaling (TS): the ability to generate temporal sequences (e.g., motor actions) at different speeds. However, it is largel…
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that proce…
Reinforcement LearningTest-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers
End-to-end ASR systems typically use fixed-depth acoustic encoders at inference, making it difficult to trade additional test-time computation for improved recognition without training a larger model. A natural approach …
Speech RecognitionStabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
Looped Language Models (LoopLMs) enable efficient latent reasoning through depth recurrence, yet exhibit unreliable test-time scaling behavior: performance often peaks at a certain iteration depth and then collapses with…
Mathematical Reasoning