Syntax-Based Attention Masking for Neural Machine Translation
We present a simple method for extending transformers to source-side trees. We define a number of masks that limit self-attention based on relationships among tree nodes, and we allow each attention head to learn which mask or masks to use. On translation from English to various low-resource languages, and translation in both directions between English and German, our method always improves over simple linearization of the source-side parse tree and almost always improves over a sequence-to-sequence baseline, by up to +2.1 BLEU.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Semantics-aware Attention Improves Neural Machine Translation
The integration of syntactic structure into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel par…
DecoderMachine TranslationTranslationSemantics-aware Attention Improves Neural Machine Translation
The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel pa…
DecoderMachine TranslationTranslationSyntax-Directed Attention for Neural Machine Translation
Attention mechanism, including global attention and local attention, plays a key role in neural machine translation (NMT). Global attention attends to all source words for word prediction. In comparison, local attention …
Machine TranslationNMTTranslationSyntax-Aware Complex-Valued Neural Machine Translation
Syntax has been proven to be remarkably effective in neural machine translation (NMT). Previous models obtained syntax information from syntactic parsing tools and integrated it into NMT models to improve translation per…
DecoderMachine TranslationNMTTranslationTop-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing
The addition of syntax-aware decoding in Neural Machine Translation (NMT) systems requires an effective tree-structured neural network, a syntax-aware attention model and a language generation model that is sensitive to …
Constituency ParsingDecoderDependency ParsingMachine Translation+4