Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA
We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA). Our model combines two original contributions”:" A multi-modal fast-learning feature fusion (FLF) block and a mechanism that uses self-attended language features to separately guide neural attention on both static and dynamic visual features extracted from individual video frames and short video clips. When trained from scratch, VLCN achieves competitive results with the state of the art on both MSVD-QA and MSRVTT-QA with 38.06% and 36.01% test accuracies, respectively. Through an ablation study, we further show that FLF improves generalization across different VideoQA datasets and performance for question types that are notoriously challenging in current datasets, such as long questions that require deeper reasoning as well as questions with rare answers.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVideo Question AnsweringSimilar Papers 제목 키워드 기반
Unlocking Financial Insights: An advanced Multimodal Summarization with Multimodal Output Framework for Financial Advisory Videos
The dynamic propagation of social media has broadened the reach of financial advisory content through podcast videos, yet extracting insights from lengthy, multimodal segments (30-40 minutes) remains challenging. We intr…
Speaker DiarizationAttention-Based Multimodal Fusion for Video Description
Currently successful methods for video description are based on encoder-decoder sentence generation using recur-rent neural networks (RNNs). Recent work has shown the advantage of integrating temporal and/or spatial atte…
DecoderSentenceVideo DescriptionMicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues
Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal emotion recognition capabilities, integrating multimodal cues from visual, acoustic, and linguistic contexts in the video to recognize huma…
Emotion RecognitionMultimodal Emotion RecognitionFV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition Inference
In the latest social networks, more and more people prefer to express their emotions in videos through text, speech, and rich facial expressions. Multimodal video emotion analysis techniques can help understand users' in…
Emotion RecognitionMultimodal Emotion RecognitionVideo Emotion RecognitionMobile-VideoGPT: Fast and Accurate Video Understanding Language Model
Video understanding models often struggle with high computational requirements, extensive parameter counts, and slow inference speed, making them inefficient for practical use. To tackle these challenges, we propose Mobi…
EgoSchemaLanguage ModelingLanguage ModellingMVBench+2