Control Image Captioning Spatially and Temporally
Generating image captions with user intention is an emerging need. The recently published Localized Narratives dataset takes mouse traces as another input to the image captioning task, which is an intuitive and efficient way for a user to control what to describe in the image. However, how to effectively employ traces to improve generation quality and controllability is still under exploration. This paper aims to solve this problem by proposing a novel model called LoopCAG, which connects Contrastive constraints and Attention Guidance in a Loop manner, engaged explicit spatial and temporal constraints to the generating process. Precisely, each generated sentence is temporally aligned to the corresponding trace sequence through a contrastive learning strategy. Besides, each generated text token is supervised to attend to the correct visual objects under heuristic spatial attention guidance. Comprehensive experimental results demonstrate that our LoopCAG model learns better correspondence among the three modalities (vision, language, and traces) and achieves SOTA performance on trace-controlled image captioning task. Moreover, the controllability and explainability of LoopCAG are validated by analyzing spatial and temporal sensitivity during the generation process.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage CaptioningSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Human Attention in Image Captioning: Dataset and Analysis
In this work, we present a novel dataset consisting of eye movements and verbal descriptions recorded synchronously over images. Using this data, we study the differences in human attention during free-viewing and image …
Image CaptioningImage DescriptionSentenceTask 2TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning
Traffic video description and analysis have received much attention recently due to the growing demand for efficient and reliable urban surveillance systems. Most existing methods only focus on locating traffic event seg…
Dense Video CaptioningDescriptiveLanguage ModelingLanguage Modelling+2CapOnImage: Context-driven Dense-Captioning on Image
Existing image captioning systems are dedicated to generating narrative captions for images, which are spatially detached from the image in presentation. However, texts can also be used as decorations on the image to hig…
Dense CaptioningDiversityImage CaptioningArea Attention
Existing attention mechanisms are trained to attend to individual items in a collection (the memory) with a predefined, fixed granularity, e.g., a word token or an image grid. We propose area attention: a way to attend t…
Image CaptioningMachine TranslationTranslationA-CAP: Anticipation Captioning with Commonsense Knowledge
Humans possess the capacity to reason about the future based on a sparse collection of visual cues acquired over time. In order to emulate this ability, we introduce a novel task called Anticipation Captioning, which gen…
Image CaptioningLanguage ModelingLanguage ModellingVisual Storytelling