Towards Multimodal Simultaneous Neural Machine Translation
Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages. This task is significantly more challenging than the general full sentence translation because of the shortage of input information during decoding. To alleviate this shortage, we propose multimodal simultaneous neural machine translation (MSNMT), which leverages visual information as an additional modality. Our experiments with the Multi30k dataset showed that MSNMT significantly outperforms its text-only counterpart in more timely translation situations with low latency. Furthermore, we verified the importance of visual information during decoding by performing an adversarial evaluation of MSNMT, where we studied how models behaved with incongruent input modality and analyzed the effect of different word order between source and target languages.
Code (1)
Tasks
Machine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation
This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual…
Machine Translationreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Supervised Visual Attention for Simultaneous Multimodal Machine Translation
Recently, there has been a surge in research in multimodal machine translation (MMT), where additional modalities such as images are used to improve translation quality of textual systems. A particular use for such multi…
Machine TranslationMultimodal Machine TranslationSentenceTranslationMultimodal Simultaneous Machine Translation
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. Therefore, translation has to start with an incomplete …
Machine TranslationTranslationSimultaneous Machine Translation with Visual Context
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. The translation thus has to start with an incomplete so…
Machine TranslationTranslationMultimodal Attention for Neural Machine Translation
The attention mechanism is an important part of the neural machine translation (NMT) where it was reported to produce richer source representation compared to fixed-length encoding sequence-to-sequence models. Recently, …
Image CaptioningMachine TranslationNMTTranslation