Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement
With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation. However, most of the previous methods combine layers in a static fashion in that their aggregation strategy is independent of specific hidden states. Inspired by recent progress on capsule networks, in this paper we propose to use routing-by-agreement strategies to aggregate layers dynamically. Specifically, the algorithm learns the probability of a part (individual layer representations) assigned to a whole (aggregated representations) in an iterative way and combines parts accordingly. We implement our algorithm on top of the state-of-the-art neural machine translation model TRANSFORMER and conduct experiments on the widely-used WMT14 English-German and WMT17 Chinese-English translation datasets. Experimental results across language pairs show that the proposed approach consistently outperforms the strong baseline model and a representative static aggregation model.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Capsule-Transformer for Neural Machine Translation
Transformer hugely benefits from its key design of the multi-head self-attention network (SAN), which extracts information from various perspectives through transforming the given input into different subspaces. However,…
Machine TranslationTranslationTowards Linear Time Neural Machine Translation with Capsule Networks
In this study, we first investigate a novel capsule network with dynamic routing for linear time Neural Machine Translation (NMT), referred as \textsc{CapsNMT}. \textsc{CapsNMT} uses an aggregation mechanism to map the s…
Machine TranslationNMTSentenceTranslationInformation Aggregation via Dynamic Routing for Sequence Encoding
While much progress has been made in how to encode a text sequence into a sequence of vectors, less attention has been paid to how to aggregate these preceding vectors (outputs of RNN/CNN) into fixed-size encoding vector…
Sentiment Analysistext-classificationText ClassificationInformation Aggregation for Multi-Head Attention with Routing-by-Agreement
Multi-head attention is appealing for its ability to jointly extract different types of information from multiple representation subspaces. Concerning the information aggregation, a common practice is to use a concatenat…
Machine TranslationTranslationResidual Tree Aggregation of Layers for Neural Machine Translation
Although attention-based Neural Machine Translation has achieved remarkable progress in recent layers, it still suffers from issue of making insufficient use of the output of each layer. In transformer, it only uses the …
DecoderMachine TranslationTranslation