Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models
In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models' ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models' parameters.
Code (1)
Tasks
Machine TranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity
Despite an in-principle understanding of self-attention matrix operations in Transformer language models (LMs), it remains unclear precisely how these operations map onto interpretable computations or functions--and how …
Word Sense DisambiguationPragmatic constraints and pronoun reference disambiguation: the possible and the impossible
Pronoun disambiguation in understanding text and discourse often requires the application of both general pragmatic knowledge and context-specific information. In AI and linguistics research, this has mostly been studied…
Do Context-Aware Translation Models Pay the Right Attention?
Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolu…
Machine TranslationTranslationCausal interventions expose implicit situation models for commonsense language understanding
Accounts of human language processing have long appealed to implicit ``situation models'' that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent tran…
World KnowledgeAn Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation
Recent work has shown that the encoder-decoder attention mechanisms in neural machine translation (NMT) are different from the word alignment in statistical machine translation. In this paper, we focus on analyzing encod…
DecoderMachine TranslationNMTTranslation+2