Single-Headed Attention

2000년 도입 · 논문 6편에서 사용

Single-Headed Attention is a single-headed attention module used in the SHA-RNN language model. The principle design reasons for single-headedness were simplicity (avoiding running out of memory) and scepticism about the benefits of using multiple heads.

출처: Single Headed Attention RNN: Stop Thinking With Your Head

소개 논문: Single Headed Attention RNN: Stop Thinking With Your Head