STOP: Spatiotemporal Orthogonal Propagation for Weight-Threshold-Leakage Synergistic Training of Deep Spiking Neural Networks
The prevailing of artificial intelligence-of-things calls for higher energy-efficient edge computing paradigms, such as neuromorphic agents leveraging brain-inspired spiking neural network (SNN) models based on spatiotemporally sparse binary spikes. However, the lack of efficient and high-accuracy deep SNN learning algorithms prevents them from practical edge deployments at a strictly bounded cost. In this paper, we propose the spatiotemporal orthogonal propagation (STOP) algorithm to tackle this challenge. Our algorithm enables fully synergistic learning of synaptic weights as well as firing thresholds and leakage factors in spiking neurons to improve SNN accuracy, in a unified temporally-forward trace-based framework to mitigate the huge memory requirement for storing neural states across all time-steps in the forward pass. Characteristically, the spatially-backward neuronal errors and temporally-forward traces propagate orthogonally to and independently of each other, substantially reducing computational complexity. Our STOP algorithm obtained high recognition accuracies of 94.84%, 74.92%, 98.26% and 77.10% on the CIFAR-10, CIFAR-100, DVS-Gesture and DVS-CIFAR10 datasets with adequate deep convolutional SNNs of VGG-11 or ResNet-18 structures. Compared with other deep SNN training algorithms, our method is more plausible for edge intelligent scenarios where resources are limited but high-accuracy in-situ learning is desired.
Code (0)
등록된 구현이 없습니다.
Tasks
Edge-computingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Instance-dependent Early Stopping
In machine learning practice, early stopping has been widely used to regularize models and can save computational costs by halting the training process when the model's performance on a validation set stops improving. Ho…
Transfer LearningGeneralized BackPropagation, Étude De Cas: Orthogonality
This paper introduces an extension of the backpropagation algorithm that enables us to have layers with constrained weights in a deep network. In particular, we make use of the Riemannian geometry and optimization techni…
Dimensionality ReductionFine-Grained Image ClassificationGeneral Classificationimage-classification+1On orthogonality and learning RNNs with long term dependencies
It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue as…
On orthogonality and learning recurrent networks with long term dependencies
It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associa…
GAIT-prop: A biologically plausible learning rule derived from backpropagation of error
Traditional backpropagation of error, though a highly successful algorithm for learning in artificial neural network models, includes features which are biologically implausible for learning in real neural circuits. An a…