SHA-RNN

2000년 도입 · 논문 2편에서 사용

SHA-RNN, or Single Headed Attention RNN, is a recurrent neural network, and language model when combined with an embedding input and softmax classifier, based on a core LSTM component and a single-headed attention module. Other design choices include a Boom feedforward layer and the use of layer normalization. The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).

출처: Single Headed Attention RNN: Stop Thinking With Your Head

소개 논문: Single Headed Attention RNN: Stop Thinking With Your Head