Semantic Video Entity Linking Based on Visual Content and Metadata
Video entity linking, which connects online videos to the related entities in a semantic knowledge base, can enable a wide variety of video based applications including video retrieval and video recommendation. Most existing systems for video entity linking rely on video metadata. In this paper, we propose to exploit video visual content to improve video entity linking. In the proposed framework, videos are first linked to entity candidates using a text-based method. Next, the entity candidates are verified and reranked according to visual content. In order to properly handle large variations in visual content matching, we propose to use Multiple Instance Metric Learning to learn a "set to sequence'' metric for this specific matching problem. To evaluate the proposed framework, we collect and annotate 1912 videos crawled from the YouTube open API. Experiment results have shown consistent gains by the proposed framework over several strong baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Entity LinkingMetric LearningRetrievalVideo RetrievalSimilar Papers 제목 키워드 기반
OVEL: Large Language Model as Memory Manager for Online Video Entity Linking
In recent years, multi-modal entity linking (MEL) has garnered increasing attention in the research community due to its significance in numerous multi-modal applications. Video, as a popular means of information transmi…
Entity LinkingLanguage ModelingLanguage ModellingLarge Language Model+1Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking
Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, …
Entity LinkingDWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking
Multimodal entity linking (MEL) aims to utilize multimodal information (usually textual and visual information) to link ambiguous mentions to unambiguous entities in knowledge base. Current methods facing main issues: (1…
Contrastive LearningEntity LinkingVisual Named Entity Linking: A New Dataset and A Baseline
Visual Entity Linking (VEL) is a task to link regions of images with their corresponding entities in Knowledge Bases (KBs), which is beneficial for many computer vision tasks such as image retrieval, image caption, and v…
Entity LinkingImage RetrievalQuestion AnsweringRetrieval+2Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning
Existing sports video captioning methods often focus on the action yet overlook player identities, limiting their applicability. Although some methods integrate extra information to generate identity-aware descriptions, …
Information ExtractionVideo Captioning