Rotary Embeddings

2000년 도입 · 논문 8편에서 사용

Rotary Position Embedding, or RoPE, is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and capability of equipping the linear self-attention with relative position encoding.

출처: RoFormer: Enhanced Transformer with Rotary Position Embedding

소개 논문: RoFormer: Enhanced Transformer with Rotary Position Embedding