Information-theoretic locality properties of natural language
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Locality in Image Diffusion Models Emerges from Data Statistics
Recent work has shown that the generalization ability of image diffusion models arises from the locality properties of the trained neural network. In particular, when denoising a particular pixel, the model relies on a l…
Language Re-generation: An investigation into information locality effects on reconstruction
Information locality, the tendency for syntactically related words to appear close together, shapes both human language processing and language model learning. While prior work has examined whether language models can ac…
The Locality and Symmetry of Positional Encodings
Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to la…
SentenceOn the Locality of the Natural Gradient for Deep Learning
We study the natural gradient method for learning in deep Bayesian networks, including neural networks. There are two natural geometries associated with such learning systems consisting of visible and hidden units. One g…
Deep LearningScaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets gene…
Representation Learning