Neuronal Attention Circuit (NAC) for Representation Learning
Attention improves representation learning over RNNs, but its discrete nature limits continuous-time (CT) modeling. We introduce Neuronal Attention Circuit (NAC), a novel, biologically inspired CT-Attention mechanism that reformulates attention logit computation as the solution to a linear first-order ODE with nonlinear interlinked gates derived from repurposing C.elegans Neuronal Circuit Policies (NCPs) wiring. NAC replaces dense projections with sparse sensory gates for key-query projections and a sparse backbone network with two heads for computing content-target and learnable time-constant gates, enabling efficient adaptive dynamics. To improve efficiency and memory consumption, we implemented an adaptable subquadratic sparse Top-K pairwise concatenation mechanism that selectively curates key-query interactions. We provide rigorous theoretical guarantees, including state stability and bounded approximation errors. Empirically, we implemented NAC in diverse domains, including irregular time-series classification, lane-keeping for autonomous vehicles, and industrial prognostics. We observed that NAC matches or outperforms competing baselines in accuracy and occupies an intermediate position in runtime and memory consumption compared with several CT state-of-the-art baselines, while being interpretable at the neuron cell level.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningAutonomous VehiclesSimilar Papers 제목 키워드 기반
Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
Reliable uncertainty quantification in continuous-time (CT) representation learning remains nascent, particularly within CT attention literature. We introduce the Neuronal Stochastic Attention Circuit (NSAC), a novel bio…
Representation LearningAutonomous VehiclesNeuronal Circuit Policies
We propose an effective way to create interpretable control agents, by re-purposing the function of a biological neural circuit model, to govern simulated and real world reinforcement learning (RL) test-beds. We model th…
Reinforcement LearningReinforcement Learning (RL)Can a Compact Neuronal Circuit Policy be Re-purposed to Learn Simple Robotic Control?
We propose a neural information processing system which is obtained by re-purposing the function of a biological neural circuit model, to govern simulated and real-world control tasks. Inspired by the structure of the ne…
A Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation
Time perception is critical in our daily life. An important feature of time perception is temporal scaling (TS): the ability to generate temporal sequences (e.g., motor actions) at different speeds. However, it is largel…
Winner-Take-All as Basic Probabilistic Inference Unit of Neuronal Circuits
Experimental observations of neuroscience suggest that the brain is working a probabilistic way when computing information with uncertainty. This processing could be modeled as Bayesian inference. However, it remains unc…
AllBayesian Inference