Streaming Batch Eigenupdates for Hardware Neuromorphic Networks
Neuromorphic networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs). Though immense acceleration of the training process can be achieved by leveraging the fact that the time complexity of training does not scale with the network size, it is limited by the space complexity of stochastic gradient descent, which grows quadratically. The main objective of this work is to reduce this space complexity by using low-rank approximations of stochastic gradient descent. This low spatial complexity combined with streaming methods allows for significant reductions in memory and compute overhead, opening the doors for improvements in area, time and energy efficiency of training. We refer to this algorithm and architecture to implement it as the streaming batch eigenupdate (SBE) approach.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing
Deep State-Space Models (SSM) demonstrate state-of-the art performance on long-range sequence modeling tasks. While the recurrent structure of SSMs can be efficiently implemented as a convolution or as a parallel scan du…
State Space ModelsLanguage Modeling on a SpiNNaker 2 Neuromorphic Chip
As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for in…
Gesture RecognitionLanguage ModelingLanguage ModellingA Low-Power Streaming Speech Enhancement Accelerator For Edge Devices
Transformer-based speech enhancement models yield impressive results. However, their heterogeneous and complex structure restricts model compression potential, resulting in greater complexity and reduced hardware efficie…
Model CompressionSpeech EnhancementTowards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limite…
Reinforcement LearningContinuous ControlEvent-based backpropagation on the neuromorphic platform SpiNNaker2
Neuromorphic computing aims to replicate the brain's capabilities for energy efficient and parallel information processing, promising a solution to the increasing demand for faster and more efficient computational system…