Delving Deeper Into Astromorphic Transformers
Preliminary attempts at incorporating the critical role of astrocytes - cells that constitute more than 50\% of human brain cells - in brain-inspired neuromorphic computing remain in infancy. This paper seeks to delve deeper into various key aspects of neuron-synapse-astrocyte interactions to mimic self-attention mechanisms in Transformers. The cross-layer perspective explored in this work involves bioplausible modeling of Hebbian and presynaptic plasticities in neuron-astrocyte networks, incorporating effects of non-linearities and feedback along with algorithmic formulations to map the neuron-astrocyte computations to self-attention mechanism and evaluating the impact of incorporating bio-realistic effects from the machine learning application side. Our analysis on sentiment and image classification tasks (IMDB and CIFAR10 datasets) highlights the advantages of Astromorphic Transformers, offering improved accuracy and learning speed. Furthermore, the model demonstrates strong natural language generation capabilities on the WikiText-2 dataset, achieving better perplexity compared to conventional models, thus showcasing enhanced generalization and stability across diverse machine learning tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Delving Deeper into Cross-lingual Visual Question Answering
Visual question answering (VQA) is one of the crucial vision-and-language tasks. Yet, existing VQA research has mostly focused on the English language, due to a lack of suitable evaluation resources. Previous work on cro…
Inductive BiasQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Radical Embedding: Delving Deeper to Chinese Radicals
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework…
DecoderInstance SegmentationPanoptic SegmentationSegmentationAstromorphic Self-Repair of Neuromorphic Hardware Systems
While neuromorphic computing architectures based on Spiking Neural Networks (SNNs) are increasingly gaining interest as a pathway toward bio-plausible machine learning, attention is still focused on computational units l…
Transformer for Object Re-Identification: A Survey
Object Re-identification (Re-ID) aims to identify specific objects across different times and scenes, which is a widely researched task in computer vision. For a prolonged period, this field has been predominantly driven…
ObjectSurvey