Learning to Answer Questions in Dynamic Audio-Visual Scenarios

2022-03-26 · CVPR 2022 1 · Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, Di Hu

In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos. The problem requires comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes. To benchmark this task and facilitate our study, we introduce a large-scale MUSIC-AVQA dataset, which contains more than 45K question-answer pairs covering 33 different question templates spanning over different modalities and question types. We develop several baselines and introduce a spatio-temporal grounded audio-visual network for the AVQA problem. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-, V-, and AVQA approaches. We believe that our built dataset has the potential to serve as testbed for evaluating and promoting progress in audio-visual scene understanding and spatio-temporal reasoning. Code and dataset: http://gewu-lab.github.io/MUSIC-AVQA/

📄 PDF Abstract BibTeX arXiv:2203.14072

Code (1)

GeWu-Lab/MUSIC-AVQA 공식 구현 pytorch

Tasks

audio-visual learningAudio-visual Question AnsweringAudio-Visual Question Answering (AVQA)AUDIO-VISUAL QUESTION ANSWERING (MUSIC-AVQA-v2.0)Question Answering Scene Understanding Visual Question Answering Visual Question Answering (VQA)

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

2024-03-07 · Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie 외

This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audio-visual components. Although existing Multimodal Large Language Models (MLLMs) can respond to aud…

Audio-visual Question Answering Audio-Visual Question Answering (AVQA)Language Modeling Language Modelling+6

Boosting Audio Visual Question Answering via Key Semantic-Aware Cues

2024-07-30 · Guangyao Li, Henghui Du, Di Hu

The Audio Visual Question Answering (AVQA) task aims to answer questions related to various visual objects, sounds, and their interactions in videos. Such naturally multimodal videos contain rich and complex dynamic audi…

Audio-visual Question Answering Audio-Visual Question Answering (AVQA)Question Answering Visual Question Answering

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

2023-05-21 · Yuanyuan Jiang, Jianqin Yin

Audio-visual question answering (AVQA) is a challenging task that requires multistep spatio-temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for s…

Audio-visual Question Answering Audio-Visual Question Answering (AVQA)Question Answering Scene Understanding+1

AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering

2025-10-21 · Jiayu Zhang, Shuo Ye, Qilang Ye, Xun Lin 외 arxiv

Audio-Visual Question Answering (AVQA) requires models to effectively utilize both visual and auditory modalities to answer complex and diverse questions about audio-visual scenes. However, existing methods lack sufficie…

Audio-visual Question Answering

Progressive Spatio-temporal Perception for Audio-Visual Question Answering

2023-08-10 · Guangyao Li, Wenxuan Hou, Di Hu

Audio-Visual Question Answering (AVQA) task aims to answer questions about different visual objects, sounds, and their associations in videos. Such naturally multi-modal videos are composed of rich and complex dynamic au…