Autocorrelations Decay in Texts and Applicability Limits of Language Models
We show that the laws of autocorrelations decay in texts are closely related to applicability limits of language models. Using distributional semantics we empirically demonstrate that autocorrelations of words in texts decay according to a power law. We show that distributional semantics provides coherent autocorrelations decay exponents for texts translated to multiple languages. The autocorrelations decay in generated texts is quantitatively and often qualitatively different from the literary texts. We conclude that language models exhibiting Markov behavior, including large autoregressive language models, may have limitations when applied to long texts, whether analysis or generation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
States of LLM-generated Texts and Phase Transitions between them
It is known for some time that autocorrelations of words in human-written texts decay according to a power law. Recent works have also shown that the autocorrelations decay in texts generated by LLMs is qualitatively dif…
Efficient Market Dynamics: Unraveling Informational Efficiency in UK Horse Racing Betting Markets Through Betfair's Time Series Analysis
Using Betfair's time series data, an analysis of the United Kingdom (UK) horse racing market reveals an interesting paradox: a market with short tails, rapidly decaying autocorrelations, and no long-term memory. There se…
Time SeriesTime Series AnalysisStatistically efficient thinning of a Markov chain sampler
It is common to subsample Markov chain output to reduce the storage burden. Geyer (1992) shows that discarding $k-1$ out of every $k$ observations will not improve statistical efficiency, as quantified through variance i…
Optimal estimation in spatially distributed systems: how far to share measurements from?
We consider the centralized optimal estimation problem in spatially distributed systems. We use the setting of spatially invariant systems as an idealization for which concrete and detailed results are given. Such estima…
Reconstructing Gridded Data from Higher Autocorrelations
The higher-order autocorrelations of integer-valued or rational-valued gridded data sets appear naturally in X-ray crystallography, and have applications in computer vision systems, correlation tomography, correlation sp…