Generative Imagination Elevates Machine Translation
There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationMultimodal Machine TranslationSentenceTransfer LearningTranslationSimilar Papers 제목 키워드 기반
Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation
Visual information has been introduced for enhancing machine translation (MT), and its effectiveness heavily relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations.…
Language ModelingLanguage ModellingLarge Language ModelMachine Translation+3Autocorrect in the Process of Translation --- Multi-task Learning Improves Dialogue Machine Translation
Automatic translation of dialogue texts is a much needed demand in many real life scenarios. However, the currently existing neural machine translation delivers unsatisfying results. In this paper, we conduct a deep anal…
Machine TranslationMulti-Task LearningTranslationAutocorrect in the Process of Translation -- Multi-task Learning Improves Dialogue Machine Translation
Automatic translation of dialogue texts is a much needed demand in many real life scenarios. However, the currently existing neural machine translation delivers unsatisfying results. In this paper, we conduct a deep anal…
Machine TranslationMulti-Task LearningTranslationPH-Dreamer: A Physics-Driven World Model via Port-Hamiltonian Generative Dynamics
World models built on recurrent state space architectures enable efficient latent imagination, yet remain physically unstructured, producing dynamics that violate conservation and dissipative principles. We introduce a u…
Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination
Meta reinforcement learning (Meta RL) has been amply explored to quickly learn an unseen task by transferring previously learned knowledge from similar tasks. However, most state-of-the-art algorithms require the meta-tr…
Meta Reinforcement Learning