Sufficient conditions for offline reactivation in recurrent neural networks
During periods of quiescence, such as sleep, neural activity in many brain circuits resembles that observed during periods of task engagement. However, the precise conditions under which task-optimized networks can autonomously reactivate the same network states responsible for online behavior is poorly understood. In this study, we develop a mathematical framework that outlines sufficient conditions for the emergence of neural reactivation in circuits that encode features of smoothly varying stimuli. We demonstrate mathematically that noisy recurrent networks optimized to track environmental state variables using change-based sensory information naturally develop denoising dynamics, which, in the absence of input, cause the network to revisit state configurations observed during periods of online activity. We validate our findings using numerical experiments on two canonical neuroscience tasks: spatial position estimation based on self-motion cues, and head direction estimation based on angular velocity cues. Overall, our work provides theoretical support for modeling offline reactivation as an emergent consequence of task optimization in noisy neural circuits.
Code (1)
Tasks
DenoisingSimilar Papers 제목 키워드 기반
The Nonequilibrium Mechanism of Noise Enhancer synergizing with Activator in HIV Latency Reactivation
Noise-modulating chemicals can synergize with transcriptional activators in reactivating latent HIV to eliminate latent HIV reservoirs. To understand the underlying biomolecular mechanism, we investigate a previous two-g…
TranslationBayesian Sparsification of Gated Recurrent Neural Networks
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent archit…
Structured Sparsification of Gated Recurrent Neural Networks
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent archi…
Language ModelingLanguage Modellingtext-classificationText ClassificationDensely Connected $G$-invariant Deep Neural Networks with Signed Permutation Representations
We introduce and investigate, for finite groups $G$, $G$-invariant deep neural network ($G$-DNN) architectures with ReLU activation that are densely connected-- i.e., include all possible skip connections. In contrast to…
3D Object ClassificationA recurrent vision transformer shows signatures of primate visual attention
Attention is fundamental to both biological and artificial intelligence, yet research on animal attention and AI self attention remains largely disconnected. We propose a Recurrent Vision Transformer (Recurrent ViT) that…
Change Detection