deepsing: Generating Sentiment-aware Visual Stories using Cross-modal Music Translation
In this paper we propose a deep learning method for performing attributed-based music-to-image translation. The proposed method is applied for synthesizing visual stories according to the sentiment expressed by songs. The generated images aim to induce the same feelings to the viewers, as the original song does, reinforcing the primary aim of music, i.e., communicating feelings. The process of music-to-image translation poses unique challenges, mainly due to the unstable mapping between the different modalities involved in this process. In this paper, we employ a trainable cross-modal translation method to overcome this limitation, leading to the first, to the best of our knowledge, deep learning method for generating sentiment-aware visual stories. Various aspects of the proposed method are extensively evaluated and discussed using different songs.
Code (1)
Tasks
Deep LearningTranslationSimilar Papers 제목 키워드 기반
SentiStory: A Multi-Layered Sentiment-Aware Generative Model for Visual Storytelling
The visual storytelling (VIST) task aims at generating reasonable, human-like and coherent stories with the image streams as input. Although many deep learning models have achieved promising results, most of them do not …
Visual StorytellingDeepSinger: Singing Voice Synthesis with Data Mined From the Web
In this paper, we develop DeepSinger, a multi-lingual multi-singer singing voice synthesis (SVS) system, which is built from scratch using singing training data mined from music websites. The pipeline of DeepSinger consi…
SentenceSinging Voice Synthesis"My Way of Telling a Story": Persona based Grounded Story Generation
Visual storytelling is the task of generating stories based on a sequence of images. Inspired by the recent works in neural generation focusing on controlling the form of text, this paper explores the idea of generating …
DecoderStory GenerationVisual Storytelling``My Way of Telling a Story'': Persona based Grounded Story Generation
Visual storytelling is the task of generating stories based on a sequence of images. Inspired by the recent works in neural generation focusing on controlling the form of text, this paper explores the idea of generating …
DecoderStory GenerationVisual StorytellingCommonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling
Visual storytelling is a task of generating relevant and interesting stories for given image sequences. In this work we aim at increasing the diversity of the generated stories while preserving the informative content fr…
DiversityInformativenessVisual Storytelling