Event and Entity Extraction from Generated Video Captions
Annotation of multimedia data by humans is time-consuming and costly, while reliable automatic generation of semantic metadata is a major challenge. We propose a framework to extract semantic metadata from automatically generated video captions. As metadata, we consider entities, the entities' properties, relations between entities, and the video category. We employ two state-of-the-art dense video captioning models with masked transformer (MT) and parallel decoding (PVDC) to generate captions for videos of the ActivityNet Captions dataset. Our experiments show that it is possible to extract entities, their properties, relations between entities, and the video category from the generated captions. We observe that the quality of the extracted information is mainly influenced by the quality of the event localization in the video as well as the performance of the event caption generation.
Code (1)
Tasks
Caption GenerationDense Video CaptioningVideo CaptioningSimilar Papers 제목 키워드 기반
Identity-Aware Human-Object Interaction Motion Captioning
Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subje…
Motion CaptioningOptimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers
Video captioning is an essential technology to understand scenes and describe events in natural language. To apply it to real-time monitoring, a system needs not only to describe events accurately but also to produce the…
Video CaptioningImplicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning
Weakly-Supervised Dense Video Captioning (WSDVC) aims to localize and describe all events of interest in a video without requiring annotations of event boundaries. This setting poses a great challenge in accurately locat…
Dense Video CaptioningDescriptiveVideo CaptioningVideo DescriptionVideo Summarization: Towards Entity-Aware Captions
Existing popular video captioning benchmarks and models deal with generic captions devoid of specific person, place or organization named entities. In contrast, news videos present a challenging setting where the caption…
Image CaptioningVideo CaptioningVideo SummarizationWorld KnowledgeTA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors
Dense video captioning aims to interpret and describe all temporally localized events throughout an input video. Recent state-of-the-art methods leverage large language models (LLMs) to provide detailed moment descriptio…
Dense Video CaptioningMoment Retrieval