Do Neural Nets Learn Statistical Laws behind Natural Language?
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Statistical Patterns in the Equations of Physics and the Emergence of a Meta-Law of Nature
Physics, as a fundamental science, aims to understand the laws of Nature and describe them in mathematical equations. While the physical reality manifests itself in a wide range of phenomena with varying levels of comple…
Symbolic RegressionAnalysis of Deep Image Quality Models
Subjective image quality measures based on deep neural networks are very related to models of visual neuroscience. This connection benefits engineering but, more interestingly, the freedom to optimize deep networks in di…
SSIMA Selective Overview of Deep Learning
Deep learning has arguably achieved tremendous success in recent years. In simple words, deep learning uses the composition of many nonlinear functions to model the complex dependency between input features and labels. W…
Deep LearningAwes, Laws, and Flaws From Today's LLM Research
We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works based on criteria typical of what is considered good r…
EthicsLanguage ModelingLanguage ModellingLarge Language ModelLinguistic laws in biology
Linguistic laws, the common statistical patterns of human language, have been investigated by quantitative linguists for nearly a century. Recently, biologists from a range of disciplines have started to explore the prev…