Diverse and Relevant Visual Storytelling with Scene Graph Embeddings
A problem in automatically generated stories for image sequences is that they use overly generic vocabulary and phrase structure and fail to match the distributional characteristics of human-generated text. We address this problem by introducing explicit representations for objects and their relations by extracting scene graphs from the images. Utilizing an embedding of this scene graph enables our model to more explicitly reason over objects and their relations during story generation, compared to the global features from an object classifier used in previous work. We apply metrics that account for the diversity of words and phrases of generated stories as well as for reference to narratively-salient image features and show that our approach outperforms previous systems. Our experiments also indicate that our models obtain competitive results on reference-based metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityStory GenerationVisual StorytellingSimilar Papers 제목 키워드 기반
Storytelling from an Image Stream Using Scene Graphs
Visual storytelling aims at generating a story from an image stream. Most existing methods tend to represent images directly with the extracted high-level features, which is not intuitive and difficult to interpret. We a…
Story GenerationVisual StorytellingStorytelling of Photo Stream with Bidirectional Multi-thread Recurrent Neural Network
Visual storytelling aims to generate human-level narrative language (i.e., a natural paragraph with multiple sentences) from a photo streams. A typical photo story consists of a global timeline with multi-thread local st…
Video CaptioningVisual StorytellingCommonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling
Visual storytelling is a task of generating relevant and interesting stories for given image sequences. In this work we aim at increasing the diversity of the generated stories while preserving the informative content fr…
DiversityInformativenessVisual StorytellingExpressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning
Scene graph generation aims to capture the semantic elements in images by modelling objects and their relationships in a structured manner, which are essential for visual understanding and reasoning tasks including image…
Common Sense ReasoningGraph GenerationImage CaptioningImage Generation+9Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
We introduce Aether Weaver, a novel, integrated framework for multimodal narrative co-generation that overcomes limitations of sequential text-to-visual pipelines. Our system concurrently synthesizes textual narratives, …