MMTF: Multi-Modal Temporal Fusion for Commonsense Video Question Answering
Video question answering is a challenging task that requires understanding the video and question in the same context. This becomes even harder when the questions involve reasoning, such as predicting future events or explaining counterfactual events, because they need knowledge not explicitly shown. Existing methods use coarse-grained fusion of video and language features, ignoring temporal information. To address this, we propose a novel vision-text fusion module that learns the temporal context of the video and question. Our module expands question tokens along the video's temporal axis and fuses them with video features to generate new representations with local and global context. We evaluated our method on four VideoQA datasets, including MSVD-QA, NExT-QA, Causal-VidQA, and AGQA-2.0.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualQuestion AnsweringVideo Question AnsweringSimilar Papers 제목 키워드 기반
Optimizing Multitask Industrial Processes with Predictive Action Guidance
Monitoring complex assembly processes is critical for maintaining productivity and ensuring compliance with assembly standards. However, variability in human actions and subjective task preferences complicate accurate ta…
Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding
Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this paper, we deal with the fast video temporal ground…
cross-modal alignmentNatural Language QueriesMMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data
Desire is a set of human aspirations and wishes that comprise verbal and cognitive aspects that drive human feelings and behaviors, distinguishing humans from other animals. Understanding human desire has the potential t…
Emotional IntelligenceEmotion RecognitionSentiment AnalysisAll in One: Exploring Unified Video-Language Pre-training
Mainstream Video-Language Pre-training models \cite{actbert,clipbert,violet} consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heav…
AllLanguage ModellingMultiple-choiceQuestion Answering+9Improving Visual Commonsense in Language Models via Multiple Image Generation
Commonsense reasoning is fundamentally based on multimodal knowledge. However, existing large language models (LLMs) are primarily trained using textual data only, limiting their ability to incorporate essential visual i…
Common Sense ReasoningImage GenerationLanguage ModelingLanguage Modelling+2