Knowledge-Enriched Visual Storytelling
Stories are diverse and highly personalized, resulting in a large possible output space for story generation. Existing end-to-end approaches produce monotonous stories because they are limited to the vocabulary and knowledge in a single training dataset. This paper introduces KG-Story, a three-stage framework that allows the story generation model to take advantage of external Knowledge Graphs to produce interesting stories. KG-Story distills a set of representative words from the input prompts, enriches the word set by using external knowledge graphs, and finally generates stories based on the enriched word set. This distill-enrich-generate framework allows the use of external resources not only for the enrichment phase, but also for the distillation and generation phases. In this paper, we show the superiority of KG-Story for visual storytelling, where the input prompt is a sequence of five photos and the output is a short story. Per the human ranking evaluation, stories generated by KG-Story are on average ranked better than that of the state-of-the-art systems. Our code and output stories are available at https://github.com/zychen423/KE-VIST.
Code (1)
Tasks
Knowledge GraphsStory GenerationVisual StorytellingSimilar Papers 제목 키워드 기반
Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling
As a technically challenging topic, visual storytelling aims at generating an imaginary and coherent story with narrative multi-sentences from a group of relevant images. Existing methods often generate direct and rigid …
DecoderStory GenerationVisual StorytellingComprehending Spatio-temporal Data via Cinematic Storytelling using Large Language Models
Spatio-temporal data captures complex dynamics across both space and time, yet traditional visualizations are complex, require domain expertise and often fail to resonate with broader audiences. Here, we propose MapMuse,…
Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning
Visual storytelling is a task of creating a short story based on photo streams. Different from visual captions, stories contain not only factual descriptions, but also imaginary concepts that do not appear in the images.…
DiversityInformativenessRelational ReasoningVisual StorytellingMMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning
We present MMCOMET, the first multimodal commonsense knowledge graph (MMKG) that integrates physical, social, and eventive knowledge. MMCOMET extends the ATOMIC2020 knowledge graph to include a visual dimension, through …
Visual StorytellingImage CaptioningImage RetrievalKnowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling
The visual storytelling (VST) task aims at generating a reasonable and coherent paragraph-level story with the image stream as input. Different from caption that is a direct and literal description of image content, the …
AI AgentKnowledge GraphsSemantic SimilaritySemantic Textual Similarity+2