Synchronous Syntactic Attention for Transformer Neural Machine Translation
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, {``}Synchronous Syntactic Attention,{''} inspired by synchronous dependency grammars. The mechanism synchronizes source-side and target-side syntactic self-attentions by minimizing the difference between target-side self-attentions and the source-side self-attentions mapped by the encoder-decoder attention matrix. The experiments show that the proposed method improves the translation performance on WMT14 En-De, WMT16 En-Ro, and ASPEC Ja-En (up to +0.38 points in BLEU).
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhancing Machine Translation with Dependency-Aware Self-Attention
Most neural machine translation models only rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. In this work, we investigate different approaches to inc…
Machine TranslationTranslationSemantics-aware Attention Improves Neural Machine Translation
The integration of syntactic structure into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel par…
DecoderMachine TranslationTranslationSemantics-aware Attention Improves Neural Machine Translation
The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel pa…
DecoderMachine TranslationTranslationSyntax-guided Localized Self-attention by Constituency Syntactic Distance
Recent works have revealed that Transformers are implicitly learning the syntactic information in its lower layers from data, albeit is highly dependent on the quality and scale of the training data. However, learning sy…
Machine TranslationTranslationSyntactic Knowledge via Graph Attention with BERT in Machine Translation
Although the Transformer model can effectively acquire context features via a self-attention mechanism, deeper syntactic knowledge is still not effectively modeled. To alleviate the above problem, we propose Syntactic kn…
Graph AttentionMachine TranslationText GenerationTranslation