Multi-Scale Attention for Audio Question Answering
Audio question answering (AQA), acting as a widely used proxy task to explore scene understanding, has got more attention. The AQA is challenging for it requires comprehensive temporal reasoning from different scales' events of an audio scene. However, existing methods mostly extend the structures of visual question answering task to audio ones in a simple pattern but may not perform well when perceiving a fine-grained audio scene. To this end, we present a Multi-scale Window Attention Fusion Model (MWAFM) consisting of an asynchronous hybrid attention module and a multi-scale window attention module. The former is designed to aggregate unimodal and cross-modal temporal contexts, while the latter captures sound events of varying lengths and their temporal dependencies for a more comprehensive understanding. Extensive experiments are conducted to demonstrate that the proposed MWAFM can effectively explore temporal information to facilitate AQA in the fine-grained scene.Code: https://github.com/GeWu-Lab/MWAFM
Code (1)
Tasks
Audio Question AnsweringQuestion AnsweringScene UnderstandingVisual Question AnsweringSimilar Papers 제목 키워드 기반
Attention-Based Methods For Audio Question Answering
Audio question answering (AQA) is the task of producing natural language answers when a system is provided with audio and natural language questions. In this paper, we propose neural network architectures based on self-a…
Audio Question AnsweringBinary ClassificationQuestion AnsweringSaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, …
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringVisual Question AnsweringQ-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio. However, many AVQA systems rely on deeply stacked layers of self- and cross attention across te…
Audio-visual Question AnsweringAudiopedia: Audio QA with Knowledge
In this paper, we introduce Audiopedia, a novel task called Audio Question Answering with Knowledge, which requires both audio comprehension and external knowledge reasoning. Unlike traditional Audio Question Answering (…
Audio Question AnsweringEntity LinkingQuestion AnsweringProgressive Spatio-temporal Perception for Audio-Visual Question Answering
Audio-Visual Question Answering (AVQA) task aims to answer questions about different visual objects, sounds, and their associations in videos. Such naturally multi-modal videos are composed of rich and complex dynamic au…
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringVisual Question Answering