Content-based Attention

2000년 도입 · 논문 33편에서 사용

Content-based attention is an attention mechanism based on cosine similarity: $$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = \cos\left[\textbf{h}\_{i};\textbf{s}\_{j}\right] $$ It was utilised in Neural Turing Machines as part of the Addressing Mechanism. We produce a normalized attention weighting by taking a softmax over these attention alignment scores.

출처: Neural Turing Machines

소개 논문: Neural Turing Machines