Temporal Knowledge-Aware Image Captioning
Contextualized image captioning is a task that extends beyond generating a purely visual description of the image content and aims to produce a caption that is influenced by the context and informed by the real world knowledge. In this paper, we present an approach to knowledge-aware image captioning, with a specific focus on the temporal domain. We propose a way to identify relevant information in external data sources, such as geographic databases and common knowledge bases, and then encode it in a way that is most useful for the captioning network. We develop an end-to-end caption generation system that incorporates external knowledge into the captioning process at several stages. The system is trained and tested on our novel temporal knowledge-aware captioning dataset, achieving significant improvements over multiple baselines across standardly used metrics. We demonstrate that our approach is effective for generating highly contextualized captions with both relevant and accurate temporal facts.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationImage CaptioningWorld KnowledgeSimilar Papers 제목 키워드 기반
Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding
Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified…
Dense Video CaptioningTransfer LearningVideo CaptioningVideo GroundingKnowledge Guided Entity-aware Video Captioning and A Basketball Benchmark
Despite the recent emergence of video captioning models, how to generate the text description with specific entity names and fine-grained actions is far from being solved, which however has great applications such as bas…
DecoderVideo CaptioningProgress-Aware Video Frame Captioning
While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video c…
Image CaptioningVideo CaptioningVideo UnderstandingExplicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
Dense video captioning jointly localizes and captions salient events in untrimmed videos. Recent methods primarily focus on leveraging additional prior knowledge and advanced multi-task architectures to achieve competiti…
Dense Video CaptioningCross-Modal RetrievalSpatio-Temporal Graph for Video Captioning with Knowledge Distillation
Video captioning is a challenging task that requires a deep understanding of visual scenes. State-of-the-art methods generate captions using either scene-level or object-level information but without explicitly modeling …
Knowledge DistillationObjectVideo CaptioningVisual Grounding