Repeated Sequences Reveal Gaps between Large Language Models and Natural Language
Evaluating whether large language models (LLMs) capture the structure of natural language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text. We propose a complementary evaluation framework based on repeated subsequences. By analyzing their distribution across scales and relating it to higher-order Rényi entropies, we probe how texts reuse previously established structure under finite-length conditions. Experiments on human-written texts and length-matched GPT-generated texts show that, while power-law models can describe restricted ranges of block length, the observed entropy growth is often equally or better characterized by logarithmic--power forms. Across datasets, natural language exhibits stable entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts. In contrast, GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size. These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic that reveals systematic differences in long-range organization, distinguishing natural language from state-of-the-art LLM outputs beyond surface-level fluency.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling
Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety risk through breadth-oriented evaluation across diverse tasks. However, real-world deployment often exposes a d…
Repetitions are not all alike: distinct mechanisms sustain repetition in language models
Text generated by language models (LMs) can degrade into repetitive cycles, where identical word sequences are persistently repeated one after another. Prior research has typically treated repetition as a unitary phenome…
AllIn-Context LearningText GenerationIn-Context Learning can distort the relationship between sequence likelihoods and biological fitness
Language models have emerged as powerful predictors of the viability of biological sequences. During training these models learn the rules of the grammar obeyed by sequences of amino acids or nucleotides. Once trained, t…
In-Context LearningLanguage ModelingLanguage ModellingMasked Language ModelingSkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution
Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to…
Reinforcement LearningThe importance of exploration: Modelling site-constant foraging
Foraging site constancy, or repeated return to the same foraging location, is a foraging strategy used by many species to decrease uncertainty, but it is often unclear exactly how the foraging site is identified. Here we…