Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

2020-05-06 · ACL 2020 6 · Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander Hauptmann

Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when the images are not available at the testing time.

📄 PDF Abstract BibTeX arXiv:2005.03119

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation Translation Unsupervised Machine Translation

Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

2023-05-20 · Hao Fei, Qian Liu, Meishan Zhang, Min Zhang 외

In this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup, inference-time image-free UMMT, where the model is trained with source-text image pairs, and tested with only source…

Hallucination Machine Translation Multimodal Machine Translation Translation

Filtering Back-Translated Data in Unsupervised Neural Machine Translation

2020-12-01 · COLING 2020 8 · Jyotsana Khatri, Pushpak Bhattacharyya

Unsupervised neural machine translation (NMT) utilizes only monolingual data for training. The quality of back-translated data plays an important role in the performance of NMT systems. In back-translation, all generated…

Domain Adaptation Machine Translation NMT Sentence+1

Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings

2020-05-30 · ACL 2021 5 · Sosuke Nishikawa, Ryokan Ri, Yoshimasa Tsuruoka

Unsupervised cross-lingual word embedding (CLWE) methods learn a linear transformation matrix that maps two monolingual embedding spaces that are separately trained with monolingual corpora. This method relies on the ass…

Cross-Lingual Word Embeddings Data Augmentation Machine Translation Translation+2

Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

2019-04-04 · NAACL 2019 6 · Jiawei Wu, Xin Wang, William Yang Wang

The overreliance on large parallel corpora significantly limits the applicability of machine translation systems to the majority of language pairs. Back-translation has been dominantly used in previous approaches for uns…

Machine Translation Sentence Translation Unsupervised Machine Translation

Supervised Visual Attention for Multimodal Neural Machine Translation

2020-12-01 · COLING 2020 8 · Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya, Yutaro Omote 외

This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding region…

Machine Translation Sentence Translation