A Cheap Linear Attention Mechanism with Fast Lookups and Fixed-Size Representations

2016-09-19 · Alexandre de Brébisson, Pascal Vincent

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not encode the sequence into a fixed-size representation but instead requires to memorize all the hidden states. These two limitations restrict the use of the softmax attention mechanism to relatively small-scale applications with short sequences and few lookups per sequence. In this work we introduce a family of linear attention mechanisms designed to overcome the two limitations listed above. We show that removing the softmax non-linearity from the traditional attention formulation yields constant-time attention lookups and fixed-size representations of the attended sequences. These properties make these linear attention mechanisms particularly suitable for large-scale applications with extreme query loads, real-time requirements and memory constraints. Early experiments on a question answering task show that these linear mechanisms yield significantly better accuracy results than no attention, but obviously worse than their softmax alternative.

📄 PDF Abstract BibTeX arXiv:1609.05866

Code (0)

등록된 구현이 없습니다.

Tasks

Question Answering

Methods 이 논문이 사용한 방법론

Content-based Attention Content-based attention is an attention mechanism based on cosine similarity: $$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) =…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

2024-03-02 · Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu 외

Large language model inference on Central Processing Units (CPU) is challenging due to the vast quantities of expensive Multiply-Add (MAD) matrix operations in the attention computations. In this paper, we argue that the…

16k CPU Language Modeling Language Modelling+1

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

2026-05-31 · Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu arxiv

Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks. Existing remedies act on the write s…

From WiscKey to Bourbon: A Learned Index for Log-Structured Merge Trees

2020-05-28 · Yifan Dai, Yien Xu, Aishwarya Ganesan, Ramnatthan Alagappan 외

We introduce BOURBON, a log-structured merge (LSM) tree that utilizes machine learning to provide fast lookups. We base the design and implementation of BOURBON on empirically-grounded principles that we derive through c…

BIG-bench Machine Learning regression

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

2026-09-14 · Muchen Li, Leonid Sigal, Renjie Liao hf

Scaling large language models efficiently has motivated sparse capacity mechanisms such as Mixture-of-Experts and, more recently, conditional memory: token-indexed embedding tables that augment the backbone with cheap pa…

GhostNetV2: Enhance Cheap Operation with Long-Range Attention

2022-11-23 · Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu 외

Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, …