Modality Shifting Attention Network for Multi-modal Video Question Answering
This paper considers a network referred to as Modality Shifting Attention Network (MSAN) for Multimodal Video Question Answering (MVQA) task. MSAN decomposes the task into two sub-tasks: (1) localization of temporal moment relevant to the question, and (2) accurate prediction of the answer based on the localized moment. The modality required for temporal localization may be different from that for answer prediction, and this ability to shift modality is essential for performing the task. To this end, MSAN is based on (1) the moment proposal network (MPN) that attempts to locate the most appropriate temporal moment from each of the modalities, and also on (2) the heterogeneous reasoning network (HRN) that predicts the answer using an attention mechanism on both modalities. MSAN is able to place importance weight on the two modalities for each sub-task using a component referred to as Modality Importance Modulation (MIM). Experimental results show that MSAN outperforms previous state-of-the-art by achieving 71.13\% test accuracy on TVQA benchmark dataset. Extensive ablation studies and qualitative analysis are conducted to validate various components of the network.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringTemporal LocalizationVideo Question AnsweringSimilar Papers 제목 키워드 기반
Modality Alignment between Deep Representations for Effective Video-and-Language Learning
Video-and-Language learning, such as video question answering or video captioning, is the next challenge in the deep learning society, as it pursues the way how human intelligence perceives everyday life. These tasks req…
Question AnsweringVideo CaptioningVideo Question AnsweringCoordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos
We present an audio-visual multimodal approach for the task of zeroshot learning (ZSL) for classification and retrieval of videos. ZSL has been studied extensively in the recent past but has primarily been limited to vis…
General ClassificationGZSL Video ClassificationRetrievalMAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention
This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -- text, audio and video -- in a multimodal video. Prior work on multimodal abstract…
Abstractive Text SummarizationMultimodal Abstractive Text SummarizationText SummarizationLearnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos
With the assumption that a video dataset is multimodality annotated in which auditory and visual modalities both are labeled or class-relevant, current multimodal methods apply modality fusion or cross-modality attention…
Action RecognitionAsynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations
Understanding human intentions (e.g., emotions) from videos has received considerable attention recently. Video streams generally constitute a blend of temporal data stemming from distinct modalities, including natural l…