Multi-Granularity Self-Attention for Neural Machine Translation
Current state-of-the-art neural machine translation (NMT) uses a deep multi-head self-attention network with no explicit phrase information. However, prior work on statistical machine translation has shown that extending the basic translation unit from words to phrases has produced substantial improvements, suggesting the possibility of improving NMT performance from explicit modeling of phrases. In this work, we present multi-granularity self-attention (Mg-Sa): a neural network that combines multi-head self-attention and phrase modeling. Specifically, we train several attention heads to attend to phrases in either n-gram or syntactic formalism. Moreover, we exploit interactions among phrases to enhance the strength of structure modeling - a commonly-cited weakness of self-attention. Experimental results on WMT14 English-to-German and NIST Chinese-to-English translation tasks show the proposed approach consistently improves performance. Targeted linguistic analysis reveals that Mg-Sa indeed captures useful phrase information at various levels of granularities.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention
Natural language sentences, being hierarchical, can be represented at different levels of granularity, like words, subwords, or characters. But most neural machine translation systems require the sentence to be represent…
DecoderMachine TranslationSentenceTranslationKnowledge Graph Enhanced Neural Machine Translation via Multi-task Learning on Sub-entity Granularity
Previous studies combining knowledge graph (KG) with neural machine translation (NMT) have two problems: i) Knowledge under-utilization: they only focus on the entities that appear in both KG and training sentence pairs,…
Machine TranslationMulti-Task LearningNMTSentence+1Hie-BART: Document Summarization with Hierarchical BART
This paper proposes a new abstractive document summarization model, hierarchical BART (Hie-BART), which captures hierarchical structures of a document (i.e., sentence-word structures) in the BART model. Although the exis…
Document SummarizationMachine TranslationSentenceTranslationArea Attention
Existing attention mechanisms are trained to attend to individual items in a collection (the memory) with a predefined, fixed granularity, e.g., a word token or an image grid. We propose area attention: a way to attend t…
Image CaptioningMachine TranslationTranslationWeighted Transformer Network for Machine Translation
State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence…
Machine TranslationTranslation