Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architectures
This paper introduces a novel approach for generating videos called Synchronized Deep Recurrent Attentive Writer (Sync-DRAW). Sync-DRAW can also perform text-to-video generation which, to the best of our knowledge, makes it the first approach of its kind. It combines a Variational Autoencoder~(VAE) with a Recurrent Attention Mechanism in a novel manner to create a temporally dependent sequence of frames that are gradually formed over time. The recurrent attention mechanism in Sync-DRAW attends to each individual frame of the video in sychronization, while the VAE learns a latent distribution for the entire video at the global level. Our experiments with Bouncing MNIST, KTH and UCF-101 suggest that Sync-DRAW is efficient in learning the spatial and temporal information of the videos and generates frames with high structural integrity, and can generate videos from simple captions on these datasets. (Accepted as oral paper in ACM-Multimedia 2017)
Code (1)
Tasks
Text-to-Video GenerationVideo GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Talking Face Generation by Conditional Recurrent Adversarial Network
Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generating the talking face video with accurate lip synchronization while maintaining smooth transition of both lip and facial mov…
Constrained Lip-synchronizationFace GenerationTalking Face GenerationVideo GenerationVideo-based Music Generation
As the volume of video content on the internet grows rapidly, finding a suitable soundtrack remains a significant challenge. This thesis presents EMSYNC (EMotion and SYNChronization), a fast, free, and automatic solution…
Emotion ClassificationMusic GenerationSpeech-Synchronized Whiteboard Generation via VLM-Driven Structured Drawing Representations
Creating whiteboard-style educational videos demands precise coordination between freehand illustrations and spoken narration, yet no existing method addresses this multimodal synchronization problem with structured, rep…
Learning Asynchronous and Sparse Human-Object Interaction in Videos
Human activities can be learned from video. With effective modeling it is possible to discover not only the action labels but also the temporal structures of the activities such as the progression of the sub-activities. …
Human-Object Interaction DetectionObjectAnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic me…
Talking Head Generationtext-to-speechText to Speech