Routing Transformer

2000년 도입 · 논문 3편에서 사용

The Routing Transformer is a Transformer that endows self-attention with a sparse routing module based on online k-means. Each attention module considers a clustering of the space: the current timestep only attends to context belonging to the same cluster. In other word, the current time-step query is routed to a limited number of context through its cluster assignment.

출처: Efficient Content-Based Sparse Attention with Routing Transformers

소개 논문: Efficient Content-Based Sparse Attention with Routing Transformers