What does Attention in Neural Machine Translation Pay Attention to?
Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Locality-Sensitive Hashing for Long Context Neural Machine Translation
After its introduction the Transformer architecture quickly became the gold standard for the task of neural machine translation. A major advantage of the Transformer compared to previous architectures is the faster train…
Document Level Machine TranslationMachine TranslationNMTSentence+1Inducing Grammars with and for Neural Machine Translation
Machine translation systems require semantic knowledge and grammatical understanding. Neural machine translation (NMT) systems often assume this information is captured by an attention mechanism and a decoder that ensure…
DecoderMachine TranslationNMTTranslationTarget Foresight Based Attention for Neural Machine Translation
In neural machine translation, an attention model is used to identify the aligned source words for a target word (target foresight word) in order to select translation context, but it does not make use of any information…
Language ModelingLanguage ModellingMachine TranslationTranslationInterrogating the Explanatory Power of Attention in Neural Machine Translation
Attention models have become a crucial component in neural machine translation (NMT). They are often implicitly or explicitly used to justify the model's decision in generating a specific token but it has not yet been ri…
counterfactualMachine TranslationNMTTranslationNeural Machine Translation with Recurrent Attention Modeling
Knowing which words have been attended to in previous time steps while generating a translation is a rich source of information for predicting what words will be attended to in the future. We improve upon the attention m…
Machine TranslationTranslation