Towards Multilingual Audio-Visual Question Answering
In this paper, we work towards extending Audio-Visual Question Answering (AVQA) to multilingual settings. Existing AVQA research has predominantly revolved around English and replicating it for addressing AVQA in other languages requires a substantial allocation of resources. As a scalable solution, we leverage machine translation and present two multilingual AVQA datasets for eight languages created from existing benchmark AVQA datasets. This prevents extra human annotation efforts of collecting questions and answers manually. To this end, we propose, MERA framework, by leveraging state-of-the-art (SOTA) video, audio, and textual foundation models for AVQA in multiple languages. We introduce a suite of models namely MERA-L, MERA-C, MERA-T with varied model architectures to benchmark the proposed datasets. We believe our work will open new research directions and act as a reference benchmark for future works in multilingual AVQA.
Code (1)
Tasks
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Machine TranslationQuestion AnsweringVisual Question AnsweringSimilar Papers 제목 키워드 기반
TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering
In healthcare and medical diagnostics, Visual Question Answering (VQA) mayemergeasapivotal tool in scenarios where analysis of intricate medical images becomes critical for accurate diagnoses. Current text-based VQA syst…
Medical Visual Question AnsweringQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)EVJVQA Challenge: Multilingual Visual Question Answering
Visual Question Answering (VQA) is a challenging task of natural language processing (NLP) and computer vision (CV), attracting significant attention from researchers. English is a resource-rich language that has witness…
Language ModelingLanguage ModellingQuestion AnsweringVietnamese Multimodal Learning+3Boosting Audio Visual Question Answering via Key Semantic-Aware Cues
The Audio Visual Question Answering (AVQA) task aims to answer questions related to various visual objects, sounds, and their interactions in videos. Such naturally multimodal videos contain rich and complex dynamic audi…
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringVisual Question AnsweringSaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, …
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringVisual Question AnsweringProgressive Spatio-temporal Perception for Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) task aims to answer questions about different visual objects, sounds, and their associations in videos. Such naturally multi-modal videos are composed of rich and complex dynamic au…
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringVisual Question Answering