Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
Reliable uncertainty quantification in continuous-time (CT) representation learning remains nascent, particularly within CT attention literature. We introduce the Neuronal Stochastic Attention Circuit (NSAC), a novel biologically-inspired CT attention architecture that reformulates attention logit computation as the solution of an Ornstein-Uhlenbeck stochastic differential equation modulated by input-dependent, nonlinear interlinked gates derived from repurposed C. elegans Neuronal Circuit Policies (NCPs) wiring mechanism. It induces a Gaussian distribution over logits that propagates principled stochasticity through a logistic-normal distribution over attention weights to yield probabilistic output. A two-term objective function combining Gaussian negative log-likelihood with an epistemic-separation regularizer enforces higher predictive variance under distributional shifts and enables joint quantification of aleatoric and epistemic uncertainty. Theoretically, we provide: (i) state stability bounds; (ii) closed-form guarantees; and (iii) frozen-coefficient error approximation. Empirically, we implement NSAC in a diverse set of learning tasks including: (i) irregular CT function approximation; (ii) multivariate regression; (iii) long-range forecasting; (iv) Industry 4.0; and (v) lane-keeping of autonomous vehicles. We observe that NSAC remains competitive against several baselines in terms of accuracy and produces informative uncertainty estimates while being interpretable at the neuronal cell level.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningAutonomous VehiclesSimilar Papers 제목 키워드 기반
Winner-Take-All as Basic Probabilistic Inference Unit of Neuronal Circuits
Experimental observations of neuroscience suggest that the brain is working a probabilistic way when computing information with uncertainty. This processing could be modeled as Bayesian inference. However, it remains unc…
AllBayesian InferenceElectrically-Tunable Stochasticity for Spin-based Neuromorphic Circuits: Self-Adjusting to Variation
Energy-efficient methods are addressed for leveraging low energy barrier nanomagnetic devices within neuromorphic architectures. Using a Magnetoresistive Random Access Memory (MRAM) probabilistic device (p-bit) as the ba…
Neuronal Attention Circuit (NAC) for Representation Learning
Attention improves representation learning over RNNs, but its discrete nature limits continuous-time (CT) modeling. We introduce Neuronal Attention Circuit (NAC), a novel, biologically inspired CT-Attention mechanism tha…
Representation LearningAutonomous VehiclesSum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits
Probabilistic circuits (PCs) have become the de-facto standard for learning and inference in probabilistic modeling. We introduce Sum-Product-Attention Networks (SPAN), a new generative model that integrates probabilisti…
Neuronal Circuit Policies
We propose an effective way to create interpretable control agents, by re-purposing the function of a biological neural circuit model, to govern simulated and real world reinforcement learning (RL) test-beds. We model th…
Reinforcement LearningReinforcement Learning (RL)