Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering
Existing video question answering (video QA) models lack the capacity for deep video understanding and flexible multistep reasoning. We propose for video QA a novel model which performs dynamic multistep reasoning between questions and videos. It creates video semantic representation based on the video scene graph composed of semantic elements of the video and semantic relations among these elements. Then, it performs multistep reasoning for better answer decision between the representations of the question and the video, and dynamically integrate the reasoning results. Experiments show the significant advantage of the proposed model against previous methods in accuracy and interpretability. Against the existing state-of-the-art model, the proposed model dramatically improves more than 4\%/3.1\%/2\% on the three widely used video QA datasets, MSRVTT-QA, MSRVTT multi-choice, and TGIF-QA, and displays better interpretability by backtracing along with the attention mechanisms to the video scene graphs.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVideo Question AnsweringVideo UnderstandingSimilar Papers 제목 키워드 기반
Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph
We propose OVIGo-3DHSG method - Open-Vocabulary Indoor Grounding of objects using 3D Hierarchical Scene Graph. OVIGo-3DHSG represents an extensive indoor environment over a Hierarchical Scene Graph derived from sequences…
Spatial ReasoningTarget-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios
Audio-visual question answering (AVQA) is a challenging task that requires multistep spatio-temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for s…
Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question AnsweringScene Understanding+1Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
Explanation and high-order reasoning capabilities are crucial for real-world visual question answering with diverse levels of inference complexity (e.g., what is the dog that is near the girl playing with?) and important…
Question AnsweringTripletVisual Question AnsweringVisual Question Answering (VQA)Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs
Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activ…
(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering
Spatio-temporal scene-graph approaches to video-based reasoning tasks, such as video question-answering (QA), typically construct such graphs for every video frame. These approaches often ignore the fact that videos are …
Question AnsweringSpatio-temporal Scene GraphsVideo Question Answering