Leveraging multimodal explanatory annotations for video interpretation with Modality Specific Dataset
We examine the impact of concept-informed supervision on multimodal video interpretation models using MOByGaze, a dataset containing human-annotated explanatory concepts. We introduce Concept Modality Specific Datasets (CMSDs), which consist of data subsets categorized by the modality (visual, textual, or audio) of annotated concepts. Models trained on CMSDs outperform those using traditional legacy training in both early and late fusion approaches. Notably, this approach enables late fusion models to achieve performance close to that of early fusion models. These findings underscore the importance of modality-specific annotations in developing robust, self-explainable video models and contribute to advancing interpretable multimodal learning in complex video analysis.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
Multimodal Large Language Models (MLLMs) have made substantial progress in recent years. However, their rigorous evaluation within specialized domains like finance is hindered by the absence of datasets characterized by …
Mathematical ReasoningViMoNet: A Multimodal Vision-Language Framework for Human Behavior Understanding from Motion and Video
This study investigates the use of large language models (LLMs) for human behavior understanding by jointly leveraging motion and video data. We argue that integrating these complementary modalities is essential for capt…
A Large-scale Interpretable Multi-modality Benchmark for Facial Image Forgery Localization
Image forgery localization, which centers on identifying tampered pixels within an image, has seen significant advancements. Traditional approaches often model this challenge as a variant of image segmentation, treating …
Face SwappingImage SegmentationLarge Language ModelMultimodal Large Language Model+1Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM
Towards open-ended Video Anomaly Detection (VAD), existing methods often exhibit biased detection when faced with challenging or unseen events and lack interpretability. To address these drawbacks, we propose Holmes-VAD,…
Anomaly DetectionAnomaly LocalizationLanguage ModelingLanguage Modelling+3LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
Recent advancements in large multimodal models (LMMs) have driven substantial progress in both text-to-video (T2V) generation and video-to-text (V2T) interpretation tasks. However, current AI-generated videos (AIGVs) sti…
BenchmarkingQuestion AnsweringText-to-Video GenerationVideo Alignment+1