An Analysis of Neural Language Modeling at Multiple Scales
Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.
Code (12)
Tasks
GPULanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
ST-Hyper: Learning High-Order Dependencies Across Multiple Spatial-Temporal Scales for Multivariate Time Series Forecasting
In multivariate time series (MTS) forecasting, many deep learning based methods have been proposed for modeling dependencies at multiple spatial (inter-variate) or temporal (intra-variate) scales. However, existing metho…
Multivariate Time Series ForecastingMulti-scale Transformer Language Models
We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias to handle the hierarchical nature of lan…
Inductive BiasLanguage ModelingLanguage ModellingMulti-Timescale Modeling of Human Behavior
In recent years, the role of artificially intelligent (AI) agents has evolved from being basic tools to socially intelligent agents working alongside humans towards common goals. In such scenarios, the ability to predict…
AI AgentMinecraftvalidNatural selection at multiple scales
Natural selection acts on traits at different scales, often with opposing consequences. This article identifies the particular forces that act at each scale and how those forces combine to determine the overall evolution…
Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretra…
Continual Pretraining