Context-Aware Group Captioning via Self-Attention and Contrastive Features
While image captioning has progressed rapidly, existing works focus mainly on describing single images. In this paper, we introduce a new task, context-aware group captioning, which aims to describe a group of target images in the context of another group of related reference images. Context-aware group captioning requires not only summarizing information from both the target and reference image group but also contrasting between them. To solve this problem, we propose a framework combining self-attention mechanism with contrastive feature construction to effectively summarize common information from each image group while capturing discriminative information between them. To build the dataset for this task, we propose to group the images and generate the group captions based on single image captions using scene graphs matching. Our datasets are constructed on top of the public Conceptual Captions dataset and our new Stock Captions dataset. Experiments on the two datasets show the effectiveness of our method on this new task. Related Datasets and code are released at https://lizw14.github.io/project/groupcap .
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningSimilar Papers 제목 키워드 기반
Geometry Attention Transformer with Position-aware LSTMs for Image Captioning
In recent years, transformer structures have been widely applied in image captioning with impressive performance. For good captioning results, the geometry and position relations of different visual objects are often tho…
DecoderImage CaptioningPositionNormalized and Geometry-Aware Self-Attention Network for Image Captioning
Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a…
Image CaptioningMachine TranslationQuestion AnsweringTranslation+3Bi-directional Contextual Attention for 3D Dense Captioning
3D dense captioning is a task involving the localization of objects and the generation of descriptions for each object in a 3D scene. Recent approaches have attempted to incorporate contextual information by modeling rel…
3D dense captioningAttributeCaption GenerationDense Captioning+1Exploring Group Video Captioning with Efficient Relational Approximation
Current video captioning efforts most focus on describing a single video while the need for captioning videos in groups has increased considerably. In this study, we propose a new task, group video captioning, which …
Video CaptioningSACT: Self-Aware Multi-Space Feature Composition Transformer for Multinomial Attention for Video Captioning
Video captioning works on the two fundamental concepts, feature detection and feature composition. While modern day transformers are beneficial in composing features, they lack the fundamental problems of selecting and u…
Dense Video CaptioningVideo Captioning