Hierarchical Context-aware Network for Dense Video Event Captioning
Dense video event captioning aims to generate a sequence of descriptive captions for each event in a long untrimmed video. Video-level context provides important information and facilities the model to generate consistent and less redundant captions between events. In this paper, we introduce a novel Hierarchical Context-aware Network for dense video event captioning (HCN) to capture context from various aspects. In detail, the model leverages local and global context with different mechanisms to jointly learn to generate coherent captions. The local context module performs full interaction between neighbor frames and the global context module selectively attends to previous or future events. According to our extensive experiment on both Youcook2 and Activitynet Captioning datasets, the video-level HCN model outperforms the event-level context-agnostic model by a large margin. The code is available at https://github.com/KirkGuo/HCN.
Code (1)
Tasks
DescriptiveSimilar Papers 제목 키워드 기반
Question-Answering Dense Video Events
This paper presents question-answering on dense video events, a novel task that answers and grounds dense-event questions in long videos, thus challenging MLLMs to faithfully comprehend and reason about multiple events o…
BenchmarkingQuestion AnsweringZero-Shot Video Question AnswerExplicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
Dense video captioning jointly localizes and captions salient events in untrimmed videos. Recent methods primarily focus on leveraging additional prior knowledge and advanced multi-task architectures to achieve competiti…
Dense Video CaptioningCross-Modal RetrievalJoint Event Detection and Description in Continuous Video Streams
Dense video captioning is a fine-grained video understanding task that involves two sub-problems: localizing distinct events in a long video stream, and generating captions for the localized events. We propose the Joint …
Dense CaptioningDense Video CaptioningEvent DetectionVideo Captioning+1Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos
Contextual reasoning is essential to understand events in long untrimmed videos. In this work, we systematically explore different captioning models with various contexts for the dense-captioning events in video task, wh…
Dense CaptioningDense Video CaptioningDiversityVideo CaptioningSemantic-Aware Pretraining for Dense Video Captioning
This report describes the details of our approach for the event dense-captioning task in ActivityNet Challenge 2021. We present a semantic-aware pretraining method for dense video captioning, which empowers the learned f…
Dense CaptioningDense Video CaptioningVideo Captioning