Leveraging Modality Tags for Enhanced Cross-Modal Video Retrieval
Video retrieval requires aligning visual content with corresponding natural language descriptions. In this paper, we introduce Modality Auxiliary Concepts for Video Retrieval (MAC-VR), a novel approach that leverages modality-specific tags -- automatically extracted from foundation models -- to enhance video retrieval. We propose to align modalities in a latent space, along with learning and aligning auxiliary latent concepts, derived from the features of a video and its corresponding caption. We introduce these auxiliary concepts to improve the alignment of visual and textual latent concepts, and so are able to distinguish concepts from one other. We conduct extensive experiments on five diverse datasets: MSR-VTT, DiDeMo, TGIF, Charades and YouCook2. The experimental results consistently demonstrate that modality-specific tags improve cross-modal alignment, outperforming current state-of-the-art methods across three datasets and performing comparably or better across the other two.
Code (0)
등록된 구현이 없습니다.
Tasks
cross-modal alignmentRetrievalVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach
Graph Foundation Models (GFMs) have achieved remarkable success in generalizing across diverse domains. However, they mainly focus on Text-Attributed Graphs (TAGs), leaving Multimodal-Attributed Graphs (MAGs) largely unt…
EMOE: Modality-Specific Enhanced Dynamic Emotion Experts
Multimodal Emotion Recognition (MER) aims to predict human emotions by leveraging multiple modalities, such as vision, acoustics, and language. However, due to the heterogeneity of these modalities, MER faces two key…
Emotion RecognitionIntent RecognitionMultimodal Emotion RecognitionMultimodal Intent RecognitionUnlocking Multi-Modal Potentials for Dynamic Text-Attributed Graph Representation
Dynamic Text-Attributed Graphs (DyTAGs) are a novel graph paradigm that captures evolving temporal edges alongside rich textual attributes. A prior approach to representing DyTAGs leverages pre-trained language models to…
Graph LearningCrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation
Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that e…
Semantic SegmentationEGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation
MultiModal Recommendation (MMR) systems have emerged as a promising solution for improving recommendation quality by leveraging rich item-side modality information, prompting a surge of diverse methods. Despite these adv…
Multimodal Recommendation