Spatio-Temporal Attention Models for Grounded Video Captioning
Automatic video captioning is challenging due to the complex interactions in dynamic real scenes. A comprehensive system would ultimately localize and track the objects, actions and interactions present in a video and generate a description that relies on temporal localization in order to ground the visual concepts. However, most existing automatic video captioning systems map from raw video data to high level textual description, bypassing localization and recognition, thus discarding potentially valuable information for content localization and generalization. In this work we present an automatic video captioning model that combines spatio-temporal attention and image classification by means of deep neural network structures based on long short-term memory. The resulting system is demonstrated to produce state-of-the-art results in the standard YouTube captioning benchmark while also offering the advantage of localizing the visual concepts (subjects, verbs, objects), with no grounding supervision, over space and time.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationTemporal LocalizationVideo CaptioningSimilar Papers 제목 키워드 기반
Spatio-Temporal Ranked-Attention Networks for Video Captioning
Generating video descriptions automatically is a challenging task that involves a complex interplay between spatio-temporal visual features and language models. Given that videos consist of spatial (frame-level) features…
Video CaptioningGrounded Video Caption Generation
We propose a new task, dataset and model for grounded video caption generation. This task unifies captioning and object grounding in video, where the objects in the caption are grounded in the video via temporally consis…
Caption GenerationImage CaptioningSpatio-Temporal Graph for Video Captioning with Knowledge Distillation
Video captioning is a challenging task that requires a deep understanding of visual scenes. State-of-the-art methods generate captions using either scene-level or object-level information but without explicitly modeling …
Knowledge DistillationObjectVideo CaptioningVisual GroundingDiverse Video Captioning by Adaptive Spatio-temporal Attention
To generate proper captions for videos, the inference needs to identify relevant concepts and pay attention to the spatial relationships between them as well as to the temporal development in the clip. Our end-to-end enc…
DecoderDiversityText GenerationVideo CaptioningObject-aware Aggregation with Bidirectional Temporal Graph for Video Captioning
Video captioning aims to automatically generate natural language descriptions of video content, which has drawn a lot of attention recent years. Generating accurate and fine-grained captions needs to not only understand …
ObjectVideo Captioning