Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models
The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions: The first one is the standard urn model which predicts that marginal frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text. The second assumption posits that the hapax rate is a simple function of the text length. Four such functions are discussed: the constant model, the Davis model, the linear model, and the logistic model. It is shown that the logistic model yields the best fit.
Code (2)
Similar Papers 제목 키워드 기반
From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
We inspect the deductive connection between the neural scaling law and Zipf's law -- two statements discussed in machine learning and quantitative linguistics. The neural scaling law describes how the cross entropy rate …
Words ranking and Hirsch index for identifying the core of the hapaxes in political texts
This paper deals with a quantitative analysis of the content of official political speeches. We study a set of about one thousand talks pronounced by the US Presidents, ranging from Washington to Trump. In particular, we…
Types, Tokens, and Hapaxes: A New Heap's Law
Heap's Law states that in a large enough text corpus, the number of types as a function of tokens grows as $N=KM^\beta$ for some free parameters $K,\beta$. Much has been written about how this result and various generali…
Do Neural Nets Learn Statistical Laws behind Natural Language?
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evi…
Deep LearningLanguage ModelingLanguage ModellingA path to natural language through tokenisation and transformers
Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear how these properties relate to the moder…