Dependency-Based Self-Attention for Transformer NMT
In this paper, we propose a new Transformer neural machine translation (NMT) model that incorporates dependency relations into self-attention on both source and target sides, dependency-based self-attention. The dependency-based self-attention is trained to attend to the modifiee for each token under constraints based on the dependency relations, inspired by Linguistically-Informed Self-Attention (LISA). While LISA is originally proposed for Transformer encoder for semantic role labeling, this paper extends LISA to Transformer NMT by masking future information on words in the decoder-side dependency-based self-attention. Additionally, our dependency-based self-attention operates at sub-word units created by byte pair encoding. The experiments show that our model improves 1.0 BLEU points over the baseline model on the WAT{'}18 Asian Scientific Paper Excerpt Corpus Japanese-to-English translation task.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationNMTSemantic Role LabelingTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Integrating Dependency Tree Into Self-attention for Sentence Representation
Recent progress on parse tree encoder for sentence representation learning is notable. However, these works mainly encode tree structures recursively, which is not conducive to parallelization. On the other hand, these w…
Computational EfficiencyRepresentation LearningSentenceSource Dependency-Aware Transformer with Supervised Self-Attention
Recently, Transformer has achieved the state-of-the-art performance on many machine translation tasks. However, without syntax knowledge explicitly considered in the encoder, incorrect context information that violates t…
Machine TranslationTranslationDistance-based Self-Attention Network for Natural Language Inference
Attention mechanism has been used as an ancillary means to help RNN or CNN. However, the Transformer (Vaswani et al., 2017) recently recorded the state-of-the-art performance in machine translation with a dramatic reduct…
Machine TranslationNatural Language InferenceSentenceTranslationDependency-Based Relative Positional Encoding for Transformer NMT
This paper proposes a new Transformer neural machine translation model that incorporates syntactic distances between two source words into the relative position representations of the self-attention mechanism. In particu…
Machine TranslationNMTPositionTranslationDeps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network
Syntax knowledge contributes its powerful strength in Neural machine translation (NMT) tasks. Early NMT works supposed that syntax details can be automatically learned from numerous texts via attention networks. However,…
Machine TranslationNMTTranslation