BigBird

2000년 도입 · 논문 16편에서 사용

BigBird is a Transformer with a sparse attention mechanism that reduces the quadratic dependency of self-attention to linear in the number of tokens. BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. In particular, BigBird consists of three main parts: - A set of $g$ global tokens attending on all parts of the sequence. - All tokens attending to a set of $w$ local neighboring tokens. - All tokens attending to a set of $r$ random tokens. This leads to a high performing attention mechanism scaling to much longer sequence lengths (8x).

출처: Big Bird: Transformers for Longer Sequences

소개 논문: Big Bird: Transformers for Longer Sequences