Generative Image Dynamics
We present an approach to modeling an image-space prior on scene motion. Our prior is learned from a collection of motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics such as trees, flowers, candles, and clothes swaying in the wind. We model this dense, long-term motion prior in the Fourier domain:given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a spectral volume, which can be converted into a motion texture that spans an entire video. Along with an image-based rendering module, these trajectories can be used for a number of downstream applications, such as turning still images into seamlessly looping videos, or allowing users to realistically interact with objects in real pictures by interpreting the spectral volumes as image-space modal bases, which approximate object dynamics.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Concept Formation and Dynamics of Repeated Inference in Deep Generative Models
Deep generative models are reported to be useful in broad applications including image generation. Repeated inference between data space and latent space in these models can denoise cluttered images and improve the quali…
Image GenerationSpontaneous Symmetry Breaking in Generative Diffusion Models
Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. In this paper, we show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides …
DiversityWonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions
WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elast…
SandScene GenerationVideo GenerationHierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
Diffusion models have achieved impressive success in high-fidelity image generation but suffer from slow sampling due to their inherently iterative denoising process. While recent one-step methods accelerate inference by…
Image GenerationPosterior sampling via Langevin dynamics based on generative priors
Posterior sampling in high-dimensional spaces using generative models holds significant promise for various applications, including but not limited to inverse problems and guided generation tasks. Despite many recent dev…
Image Restoration