Adversarial Evaluation of Multimodal Machine Translation
The promise of combining language and vision in multimodal machine translation is that systems will produce better translations by leveraging the image data. However, the evidence surrounding whether the images are useful is unconvincing due to inconsistencies between text-similarity metrics and human judgements. We present an adversarial evaluation to directly examine the utility of the image data in this task. Our evaluation tests whether systems perform better when paired with congruent images or incongruent images. This evaluation shows that only one out of three publicly available systems is sensitive to this perturbation of the data. We recommend that multimodal translation systems should be able to pass this sanity check in the future.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationMultimodal Machine Translationtext similarityTranslationSimilar Papers 제목 키워드 기반
Towards Multimodal Simultaneous Neural Machine Translation
Simultaneous translation involves translating a sentence before the speaker's utterance is completed in order to realize real-time understanding in multiple languages. This task is significantly more challenging than the…
Machine TranslationSentenceTranslationImproved English to Hindi Multimodal Neural Machine Translation
Machine translation performs automatic translation from one natural language to another. Neural machine translation attains a state-of-the-art approach in machine translation, but it requires adequate training data, whic…
Data AugmentationMachine TranslationNMTTranslationDoes Multimodality Help Human and Machine for Translation and Image Captioning?
This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural netw…
Image CaptioningImage DescriptionMachine TranslationMultimodal Machine Translation+1Multimodal Machine Translation with Reinforcement Learning
Multimodal machine translation is one of the applications that integrates computer vision and language processing. It is a unique task given that in the field of machine translation, many state-of-the-arts algorithms sti…
Image DescriptionMachine TranslationMultimodal Machine Translationreinforcement-learning+3English to Bengali Multimodal Neural Machine Translation using Transliteration-based Phrase Pairs Augmentation
Automatic translation of one natural language to another is a popular task of natural language processing. Although the deep learning-based technique known as neural machine translation (NMT) is a widely accepted machine…
Machine TranslationNMTTranslationTransliteration