Emergence of robust memory manifolds
The ability to store continuous variables in the state of a biological system (e.g. a neural network) is critical for many behaviours. Most models for implementing such a memory manifold require hand-crafted symmetries in the interactions or precise fine-tuning of parameters. We present a general principle that we refer to as {\it frozen stabilisation} (FS), which allows a family of neural networks to self-organise to a critical state exhibiting multiple memory manifolds without parameter fine-tuning or symmetries. Memory manifolds arising from FS exhibit a wide range of emergent relaxational timescales and can be used as general purpose integrators for inputs aligned with the manifold. Moreover, FS allows robust memory manifolds in small networks, and this is relevant to debates of implementing continuous attractors with a small number of neurons in light of recent experimental discoveries.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
OCTANE -- Optimal Control for Tensor-based Autoencoder Network Emergence: Explicit Case
This paper presents a novel, mathematically rigorous framework for autoencoder-type deep neural networks that combines optimal control theory and low-rank tensor methods to yield memory-efficient training and automated a…
Image DenoisingEmergence of Separable Manifolds in Deep Language Representations
Deep neural networks (DNNs) have shown much empirical success in solving perceptual tasks across various cognitive modalities. While they are only loosely inspired by the biological brain, recent studies report considera…
Generalizing Adam to Manifolds for Efficiently Training Transformers
One of the primary reasons behind the success of neural networks has been the emergence of an array of new, highly-successful optimizers, perhaps most importantly the Adam optimizer. It is widely used for training neural…
Physical IntuitionMemorization to Generalization: Emergence of Diffusion Models from Associative Memory
Hopfield networks are associative memory (AM) systems, designed for storing and retrieving patterns as local minima of an energy landscape. In the classical Hopfield model, an interesting phenomenon occurs when the amoun…
MemorizationRetrievalOn Dominant Manifolds in Reservoir Computing Networks
Understanding how training shapes the geometry of recurrent network dynamics is a central problem in time-series modeling. We study the emergence of low-dimensional dominant manifolds in the training of Reservoir Computi…