Zipf's law emerges asymptotically during phase transitions in communicative systems
Zipf's law predicts a power-law relationship between word rank and frequency in language communication systems, and is widely reported in texts yet remains enigmatic as to its origins. Computer simulations have shown that language communication systems emerge at an abrupt phase transition in the fidelity of mappings between symbols and objects. Since the phase transition approximates the Heaviside or step function, we show that Zipfian scaling emerges asymptotically at high rank based on the Laplace transform. We thereby demonstrate that Zipf's law gradually emerges from the moment of phase transition in communicative systems. We show that this power-law scaling behavior explains the emergence of natural languages at phase transitions. We find that the emergence of Zipf's law during language communication suggests that the use of rare words in a lexicon is critical for the construction of an effective communicative system at the phase transition.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Phase transitions in a decentralized graph-based approach to human language
Zipf's law establishes a scaling behavior for word-frequencies in large text corpora. The appearance of Zipfian properties in human language has been previously explained as an optimization problem for the interests of s…
Evidence of Phase Transitions in Small Transformer-Based Language Models
Phase transitions have been proposed as the origin of emergent abilities in large language models (LLMs), where new capabilities appear abruptly once models surpass critical thresholds of scale. Prior work, such as that …
Pragmatic Constraint on Distributional Semantics
This paper studies the limits of language models' statistical learning in the context of Zipf's law. First, we demonstrate that Zipf-law token distribution emerges irrespective of the chosen tokenization. Second, we show…
Relating Turing's Formula and Zipf's Law
An asymptote is derived from Turing's local reestimation formula for population frequencies, and a local reestimation formula is derived from Zipf's law for the asymptotic behavior of population frequencies. The two are …
Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition
We investigate phase transitions in a Toy Model of Superposition (TMS) using Singular Learning Theory (SLT). We derive a closed formula for the theoretical loss and, in the case of two hidden dimensions, discover that re…
Learning Theory