Linformer

2000년 도입 · 논문 17편에서 사용

Linformer is a linear Transformer that utilises a linear self-attention mechanism to tackle the self-attention bottleneck with Transformer models. The original scaled dot-product attention is decomposed into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.

출처: Linformer: Self-Attention with Linear Complexity

소개 논문: Linformer: Self-Attention with Linear Complexity