MaXM: Towards Multilingual Visual Question Answering
Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.
Code (1)
Tasks
Question AnsweringTranslationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
EVJVQA Challenge: Multilingual Visual Question Answering
Visual Question Answering (VQA) is a challenging task of natural language processing (NLP) and computer vision (CV), attracting significant attention from researchers. English is a resource-rich language that has witness…
Language ModelingLanguage ModellingQuestion AnsweringVietnamese Multimodal Learning+3A Unified Framework for Multilingual and Code-Mixed Visual Question Answering
In this paper, we propose an effective deep learning framework for multilingual and code- mixed visual question answering. The pro- posed model is capable of predicting answers from the questions in Hindi, English or Cod…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)xGQA: Cross-Lingual Visual Question Answering
Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we addres…
Cross-Lingual TransferLanguage ModelingLanguage ModellingQuestion Answering+3xGQA: Cross-Lingual Visual Question Answering
Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we addres…
Cross-Lingual TransferLanguage ModelingLanguage ModellingQuestion Answering+3Integrating Image Features with Convolutional Sequence-to-sequence Network for Multilingual Visual Question Answering
Visual Question Answering (VQA) is a task that requires computers to give correct answers for the input questions based on the images. This task can be solved by humans with ease but is a challenge for computers. The VLS…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)