NUWA-LIP: Language-Guided Image Inpainting With Defect-Free VQGAN
Language-guided image inpainting aims to fill the defective regions of an image under the guidance of text while keeping the non-defective regions unchanged. However, directly encoding the defective images is prone to have an adverse effect on the non-defective regions, giving rise to distorted structures on non-defective parts. To better adapt the text guidance to the inpainting task, this paper proposes NUWA-LIP, which involves defect-free VQGAN (DF-VQGAN) and a multi-perspective sequence-to-sequence module (MP-S2S). To be specific, DF-VQGAN introduces relative estimation to carefully control the receptive spreading, as well as symmetrical connections to protect structure details unchanged. For harmoniously embedding text guidance into the locally defective regions, MP-S2S is employed by aggregating the complementary perspectives from low-level pixels, high-level tokens as well as the text description. Experiments show that our DF-VQGAN effectively aids the inpainting process while avoiding unexpected changes in non-defective regions. Results on three open-domain benchmarks demonstrate the superior performance of our method against state-of-the-arts. Our code, datasets, and model will be made publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
Image InpaintingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NÜWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN
Language guided image inpainting aims to fill in the defective regions of an image under the guidance of text while keeping non-defective regions unchanged. However, the encoding process of existing models suffers from e…
Image InpaintingNUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos. An autoregressive ov…
Image OutpaintingText-to-Image GenerationVideo GenerationDefectFill: Realistic Defect Generation with Inpainting Diffusion Model for Visual Inspection
Developing effective visual inspection models remains challenging due to the scarcity of defect data. While image generation models have been used to synthesize defect images, producing highly realistic defects remains d…
Image GenerationStrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis
To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic …
Vector GraphicsLayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models
Graphic layout generation, a growing research field, plays a significant role in user engagement and information perception. Existing methods primarily treat layout generation as a numerical optimization task, focusing o…
Code CompletionCode GenerationLayout Generation