Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior
Colorizing grayscale images offers an engaging visual experience. Existing automatic colorization methods often fail to generate satisfactory results due to incorrect semantic colors and unsaturated colors. In this work, we propose an automatic colorization pipeline to overcome these challenges. We leverage the extraordinary generative ability of the diffusion prior to synthesize color with plausible semantics. To overcome the artifacts introduced by the diffusion prior, we apply the luminance conditional guidance. Moreover, we adopt multimodal high-level semantic priors to help the model understand the image content and deliver saturated colors. Besides, a luminance-aware decoder is designed to restore details and enhance overall visual quality. The proposed pipeline synthesizes saturated colors while maintaining plausible semantics. Experiments indicate that our proposed method considers both diversity and fidelity, surpassing previous methods in terms of perceptual realism and gain most human preference.
Code (0)
등록된 구현이 없습니다.
Tasks
ColorizationDecoderDiversityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Representations for Automatic Colorization
We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations. As many scene elements naturally appear according t…
ColorizationImage ColorizationRepresentation LearningDiffusing Colors: Image Colorization with Text Guided Diffusion
The colorization of grayscale images is a complex and subjective task with significant challenges. Despite recent progress in employing large-scale datasets with deep neural networks, difficulties with controllability an…
ColorizationImage ColorizationImproved Diffusion-based Image Colorization via Piggybacked Models
Image colorization has been attracting the research interests of the community for decades. However, existing methods still struggle to provide satisfactory colorized results given grayscale images due to a lack of human…
ColorizationImage ColorizationVanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization
Video colorization aims to transform grayscale videos into vivid color representations while maintaining temporal consistency and structural integrity. Existing video colorization methods often suffer from color bleeding…
ColorizationOptical Flow EstimationPrompt-based Consistent Video Colorization
Existing video colorization methods struggle with temporal flickering or demand extensive manual input. We propose a novel approach automating high-fidelity video colorization using rich semantic guidance derived from la…