Hybrid Self-Attention Network for Machine Translation
The encoder-decoder is the typical framework for Neural Machine Translation (NMT), and different structures have been developed for improving the translation performance. Transformer is one of the most promising structures, which can leverage the self-attention mechanism to capture the semantic dependency from global view. However, it cannot distinguish the relative position of different tokens very well, such as the tokens located at the left or right of the current token, and cannot focus on the local information around the current token either. To alleviate these problems, we propose a novel attention mechanism named Hybrid Self-Attention Network (HySAN) which accommodates some specific-designed masks for self-attention network to extract various semantic, such as the global/local information, the left/right part context. Finally, a squeeze gate is introduced to combine different kinds of SANs for fusion. Experimental results on three machine translation tasks show that our proposed framework outperforms the Transformer baseline significantly and achieves superior results over state-of-the-art NMT systems.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationNMTTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Accelerating Transformer Decoding via a Hybrid of Self-attention and Recurrent Neural Network
Due to the highly parallelizable architecture, Transformer is faster to train than RNN-based models and popularly used in machine translation tasks. However, at inference time, each output word requires all the hidden st…
DecoderKnowledge DistillationMachine TranslationTranslationTowards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons
Recent studies have shown that a hybrid of self-attention networks (SANs) and recurrent neural networks (RNNs) outperforms both individual architectures, while not much is known about why the hybrid models work. With the…
Inductive BiasMachine TranslationTranslationNeural Machine Translation Leveraging Phrase-based Models in a Hybrid Search
In this paper, we introduce a hybrid search for attention-based neural machine translation (NMT). A target phrase learned with statistical MT models extends a hypothesis in the NMT beam search when the attention of the N…
Language ModelingLanguage ModellingMachine TranslationNMT+1Synchronous Syntactic Attention for Transformer Neural Machine Translation
This paper proposes a novel attention mechanism for Transformer Neural Machine Translation, {``}Synchronous Syntactic Attention,{''} inspired by synchronous dependency grammars. The mechanism synchronizes source-side and…
DecoderMachine TranslationTranslationMulti-Granularity Self-Attention for Neural Machine Translation
Current state-of-the-art neural machine translation (NMT) uses a deep multi-head self-attention network with no explicit phrase information. However, prior work on statistical machine translation has shown that extending…
Machine TranslationNMTTranslation