Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
We study the learning dynamics and the representations emerging in Recurrent Neural Networks trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which a RNN with n neurons learns to integrate D(n) scalar signals of arbitrary duration. We show, both for linear and ReLU neurons, that its internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Embedding of Low-Dimensional Sensory Dynamics in Recurrent Networks: Implications for the Geometry of Neural Representation
Neural population activity in sensory cortex is organized on low-dimensional manifolds, but why such manifolds arise and what determines their geometry remain unclear. We model cortical populations as recurrent circuits …
Constructive Disintegration and Conditional Modes
Conditioning, the central operation in Bayesian statistics, is formalised by the notion of disintegration of measures. However, due to the implicit nature of their definition, constructing disintegrations is often diffic…
Bayesian InferenceOn Dominant Manifolds in Reservoir Computing Networks
Understanding how training shapes the geometry of recurrent network dynamics is a central problem in time-series modeling. We study the emergence of low-dimensional dominant manifolds in the training of Reservoir Computi…
Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds
Normalizing flows are generative models that provide tractable density estimation by transforming a simple distribution into a complex one. However, flows cannot directly model data supported on an unknown low-dimensiona…
Density EstimationPepSIRF: a flexible and comprehensive tool for the analysis of data from highly-multiplexed DNA-barcoded peptide assays
By coupling peptides with DNA tags (i.e., 'barcodes'), it is now possible to harness high-throughput sequencing (HTS) technologies to enable highly multiplexed peptide-based assays, which have a variety of potential appl…