VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
We propose $\textbf{VidStyleODE}$, a spatiotemporally continuous disentangled $\textbf{Vid}$eo representation based upon $\textbf{Style}$GAN and Neural-$\textbf{ODE}$s. Effective traversal of the latent space learned by Generative Adversarial Networks (GANs) has been the basis for recent breakthroughs in image editing. However, the applicability of such advancements to the video domain has been hindered by the difficulty of representing and controlling videos in the latent space of GANs. In particular, videos are composed of content (i.e., appearance) and complex motion components that require a special mechanism to disentangle and control. To achieve this, VidStyleODE encodes the video content in a pre-trained StyleGAN $\mathcal{W}_+$ space and benefits from a latent ODE component to summarize the spatiotemporal dynamics of the input video. Our novel continuous video generation process then combines the two to generate high-quality and temporally consistent videos with varying frame rates. We show that our proposed method enables a variety of applications on real videos: text-guided appearance manipulation, motion manipulation, image animation, and video interpolation and extrapolation. Project website: https://cyberiada.github.io/VidStyleODE
Code (0)
등록된 구현이 없습니다.
Tasks
Image AnimationVideo EditingVideo GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Latent Transformations via NeuralODEs for GAN-based Image Editing
Recent advances in high-fidelity semantic image editing heavily rely on the presumably disentangled latent spaces of the state-of-the-art generative models, such as StyleGAN. Specifically, recent works show that it is po…
AttributeVideo2StyleGAN: Disentangling Local and Global Variations in a Video
Image editing using a pretrained StyleGAN generator has emerged as a powerful paradigm for facial editing, providing disentangled controls over age, expression, illumination, etc. However, the approach cannot be directly…
Facial EditingStyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
One-shot talking face generation aims at synthesizing a high-quality talking face video from an arbitrary portrait image, driven by a video or an audio segment. One challenging quality factor is the resolution of the out…
Face GenerationFacial EditingMotion GenerationTalking Face Generation+1A Latent Transformer for Disentangled Face Editing in Images and Videos
High quality facial image editing is a challenging problem in the movie post-production industry, requiring a high degree of control and identity preservation. Previous works that attempt to tackle this problem may suffe…
AttributeDisentanglementPerceptually Validated Precise Local Editing for Facial Action Units with StyleGAN
The ability to edit facial expressions has a wide range of applications in computer graphics. The ideal facial expression editing algorithm needs to satisfy two important criteria. First, it should allow precise and targ…