Multimodal Machine Translation with Embedding Prediction
Multimodal machine translation is an attractive application of neural machine translation (NMT). It helps computers to deeply understand visual objects and their relations with natural languages. However, multimodal NMT systems suffer from a shortage of available training data, resulting in poor performance for translating rare words. In NMT, pretrained word embeddings have been shown to improve NMT of low-resource domains, and a search-based approach is proposed to address the rare word problem. In this study, we effectively combine these two approaches in the context of multimodal NMT and explore how we can take full advantage of pretrained word embeddings to better translate rare words. We report overall performance improvements of 1.24 METEOR and 2.49 BLEU and achieve an improvement of 7.67 F-score for rare word translation.
Code (1)
Tasks
Machine TranslationMultimodal Machine TranslationNMTPredictionTranslationWord EmbeddingsWord TranslationSimilar Papers 제목 키워드 기반
Debiasing Word Embeddings Improves Multimodal Machine Translation
In recent years, pretrained word embeddings have proved useful for multimodal neural machine translation (NMT) models to address the shortage of available datasets. However, the integration of pretrained word embeddings …
Machine TranslationMultimodal Machine TranslationNMTTranslation+1Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
In Multimodal Neural Machine Translation (MNMT), a neural model generates a translated sentence that describes an image, given the image itself and one source descriptions in English. This is considered as the multimodal…
Dense CaptioningMachine Translationobject-detectionObject Detection+3Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation
Visual Genome is a dataset connecting structured image information with English language. We present ``Hindi Visual Genome'', a multimodal dataset consisting of text and images suitable for English-Hindi multimodal machi…
Machine TranslationMultimodal Machine TranslationTranslationOn Leveraging the Visual Modality for Neural Machine Translation
Leveraging the visual modality effectively for Neural Machine Translation (NMT) remains an open problem in computational linguistics. Recently, Caglayan et al. posit that the observed gains are limited mainly due to the …
Machine TranslationMultimodal Machine TranslationNMTSentence+1Multimodal Quality Estimation for Machine Translation
We propose approaches to Quality Estimation (QE) for Machine Translation that explore both text and visual modalities for Multimodal QE. We compare various multimodality integration and fusion strategies. For both senten…
Machine TranslationSentenceTranslation