Semantically Consistent Video Inpainting with Conditional Diffusion Models
Current state-of-the-art methods for video inpainting typically rely on optical flow or attention-based approaches to inpaint masked regions by propagating visual information across frames. While such approaches have led to significant progress on standard benchmarks, they struggle with tasks that require the synthesis of novel content that is not present in other frames. In this paper, we reframe video inpainting as a conditional generative modeling problem and present a framework for solving such problems with conditional video diffusion models. We introduce inpainting-specific sampling schemes which capture crucial long-range dependencies in the context, and devise a novel method for conditioning on the known pixels in incomplete frames. We highlight the advantages of using a generative approach for this task, showing that our method is capable of generating diverse, high-quality inpaintings and synthesizing new content that is spatially, temporally, and semantically consistent with the provided context.
Code (0)
등록된 구현이 없습니다.
Tasks
Optical Flow EstimationVideo InpaintingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
3D-Consistent Image Inpainting with Diffusion Models
We address the problem of 3D inconsistency of image inpainting based on diffusion models. We propose a generative model using image pairs that belong to the same scene. To achieve the 3D-consistent and semantically coher…
DenoisingImage InpaintingInductive BiasVipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models
Recent video inpainting methods have achieved encouraging improvements by leveraging optical flow to guide pixel propagation from reference frames either in the image space or feature space. However, they would produce s…
DenoisingImage InpaintingOptical Flow EstimationVideo InpaintingImproving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network
Contemporary benchmark methods for image inpainting are based on deep generative models and specifically leverage adversarial loss for yielding realistic reconstructions. However, these models cannot be directly applied …
Generative Adversarial NetworkImage InpaintingConditional Diffusion Models for Global Precipitation Map Inpainting
Incomplete satellite-based precipitation presents a significant challenge in global monitoring. For example, the Global Satellite Mapping of Precipitation (GSMaP) from JAXA suffers from substantial missing regions due to…
Video InpaintingVideo Diffusion Models are Strong Video Inpainter
Propagation-based video inpainting using optical flow at the pixel or feature level has recently garnered significant attention. However, it has limitations such as the inaccuracy of optical flow prediction and the propa…
Optical Flow EstimationVideo Inpainting