Towards Linear Time Neural Machine Translation with Capsule Networks
In this study, we first investigate a novel capsule network with dynamic routing for linear time Neural Machine Translation (NMT), referred as \textsc{CapsNMT}. \textsc{CapsNMT} uses an aggregation mechanism to map the source sentence into a matrix with pre-determined size, and then applys a deep LSTM network to decode the target sequence from the source representation. Unlike the previous work \cite{sutskever2014sequence} to store the source sentence with a passive and bottom-up way, the dynamic routing policy encodes the source sentence with an iterative process to decide the credit attribution between nodes from lower and higher layers. \textsc{CapsNMT} has two core properties: it runs in time that is linear in the length of the sequences and provides a more flexible way to select, represent and aggregates the part-whole information of the source sentence. On WMT14 English-German task and a larger WMT14 English-French task, \textsc{CapsNMT} achieves comparable results with the state-of-the-art NMT systems. To the best of our knowledge, this is the first work that capsule networks have been empirically investigated for sequence to sequence problems.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Capsule-Transformer for Neural Machine Translation
Transformer hugely benefits from its key design of the multi-head self-attention network (SAN), which extracts information from various perspectives through transforming the given input into different subspaces. However,…
Machine TranslationTranslationEnhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation
Context modeling is essential to generate coherent and consistent translation for Document-level Neural Machine Translations. The widely used method for document-level translation usually compresses the context informati…
TranslationImproving Multi-Head Attention with Capsule Networks
Multi-head attention advances neural machine translation by working out multiple versions of attention in different subspaces, but the neglect of semantic overlapping between subspaces increases the difficulty of transla…
ClusteringMachine TranslationTranslationA quantifiable testing of global translational invariance in Convolutional and Capsule Networks
We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset. Experiments on convolutional and capsules neural networks show that both models have poor …
Data AugmentationTranslationDynamic Context-guided Capsule Network for Multimodal Machine Translation
Multimodal machine translation (MMT), which mainly focuses on enhancing text-only translation with visual features, has attracted considerable attention from both computer vision and natural language processing communiti…
DecoderMachine TranslationMultimodal Machine TranslationRepresentation Learning+1