Supervised Visual Attention for Simultaneous Multimodal Machine Translation

2022-01-23 · Veneta Haralampieva, Ozan Caglayan, Lucia Specia

Recently, there has been a surge in research in multimodal machine translation (MMT), where additional modalities such as images are used to improve translation quality of textual systems. A particular use for such multimodal systems is the task of simultaneous machine translation, where visual context has been shown to complement the partial information provided by the source sentence, especially in the early phases of translation. In this paper, we propose the first Transformer-based simultaneous MMT architecture, which has not been previously explored in the field. Additionally, we extend this model with an auxiliary supervision signal that guides its visual attention mechanism using labelled phrase-region alignments. We perform comprehensive experiments on three language directions and conduct thorough quantitative and qualitative analyses using both automatic metrics and manual inspection. Our results show that (i) supervised visual attention consistently improves the translation quality of the MMT models, and (ii) fine-tuning the MMT with supervision loss enabled leads to better performance than training the MMT from scratch. Compared to the state-of-the-art, our proposed model achieves improvements of up to 2.3 BLEU and 3.5 METEOR points.

📄 PDF Abstract BibTeX arXiv:2201.09324

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation Multimodal Machine TranslationSentenceTranslation

Supervised Visual Attention for Multimodal Neural Machine Translation

2020-12-01 · COLING 2020 8 · Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya, Yutaro Omote 외

This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding region…

Machine Translation Sentence Translation

Separating Content and Style for Unsupervised Image-to-Image Translation

2021-10-27 · Yunfei Liu, Haofei Wang, Yang Yue, Feng Lu

Unsupervised image-to-image translation aims to learn the mapping between two visual domains with unpaired samples. Existing works focus on disentangling domain-invariant content code and domain-specific style code indiv…

Diversity Image-to-Image Translation Translation Unsupervised Image-To-Image Translation

Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation

2021-02-22 · EACL 2021 2 · Julia Ive, Andy Mingren Li, Yishu Miao, Ozan Caglayan 외

This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual…

Machine Translation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Multimodal Unified Attention Networks for Vision-and-Language Interactions

2019-08-12 · Zhou Yu, Yuhao Cui, Jun Yu, DaCheng Tao 외

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approa…

Question Answering Visual Grounding Visual Question Answering Visual Question Answering (VQA)

Visual Explanations from Hadamard Product in Multimodal Deep Networks

2017-12-18 · Jin-Hwa Kim, Byoung-Tak Zhang

The visual explanation of learned representation of models helps to understand the fundamentals of learning. The attentional models of previous works used to visualize the attended regions over an image or text using the…

Question Answering Visual Question Answering Visual Question Answering (VQA)