Attention Approximates Sparse Distributed Memory

2021-11-10 · NeurIPS 2021 12 · Trenton Bricken, Cengiz Pehlevan

While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.

📄 PDF Abstract BibTeX arXiv:2111.05498

Code (1)

trentbrick/attention-approximates-sdm 공식 구현 jax

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Distributed-Memory Sparse Kernels for Machine Learning

2022-03-15 · Vivek Bharadwaj, Aydın Buluç, James Demmel

Sampled Dense Times Dense Matrix Multiplication (SDDMM) and Sparse Times Dense Matrix Multiplication (SpMM) appear in diverse settings, such as collaborative filtering, document clustering, and graph embedding. Frequentl…

BIG-bench Machine Learning Collaborative Filtering Graph Embedding

Sparse Distributed Memory is a Continual Learner

2023-03-20 · Trenton Bricken, Xander Davies, Deepak Singh, Dmitry Krotov 외

Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the pow…

Continual Learning

Prototype memory and attention mechanisms for few shot image generation

2021-09-29 · ICLR 2022 4 · Tianqin Li, Zijie Li, Andrew Luo, Harold Rockwell 외

Recent discoveries indicate that the neural codes in the primary visual cortex (V1) of macaque monkeys are complex, diverse and sparse. This leads us to ponder the computational advantages and functional role of these “g…

Image Generation Online Clustering

Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs

2026-04-17 · Jun-Liang Lin, Kamesh Madduri, Mahmut Taylan Kandemir arxiv

Graph foundation models have demonstrated remarkable adaptability across diverse downstream tasks through large-scale pretraining on graphs. However, existing implementations of the backbone model, graph transformers, ar…

MiA-Signature: Approximating Global Activation for Long-Context Understanding

2026-05-07 · Yuqing Li, Jiangnan Li, Mo Yu, Zheng Lin 외 arxiv

A growing body of work in cognitive science suggests that reportable conscious access is associated with \emph{global ignition} over distributed memory systems, while such activation is only partially accessible as indiv…

Long-Context Understanding