KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models
Recent advances in diffusion models have significantly improved text-to-image (T2I) generation, but they often struggle to balance fine-grained precision with high-level control. Methods like ControlNet and T2I-Adapter excel at following sketches by seasoned artists but tend to be overly rigid, replicating unintentional flaws in sketches from novice users. Meanwhile, coarse-grained methods, such as sketch-based abstraction frameworks, offer more accessible input handling but lack the precise control needed for detailed, professional use. To address these limitations, we propose KnobGen, a dual-pathway framework that democratizes sketch-based image generation by seamlessly adapting to varying levels of sketch complexity and user skill. KnobGen uses a Coarse-Grained Controller (CGC) module for high-level semantics and a Fine-Grained Controller (FGC) module for detailed refinement. The relative strength of these two modules can be adjusted through our knob inference mechanism to align with the user's specific needs. These mechanisms ensure that KnobGen can flexibly generate images from both novice sketches and those drawn by seasoned artists. This maintains control over the final output while preserving the natural appearance of the image, as evidenced on the MultiGen-20M dataset and a newly collected sketch dataset.
Code (1)
Tasks
Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sketch & Paint: Stroke-by-Stroke Evolution of Visual Artworks
Understanding the stroke-based evolution of visual artworks is useful for advancing artwork learning, appreciation, and interactive display. While the stroke sequence of renowned artworks remains largely unknown, formula…
ClusteringDiffMorph: Text-less Image Morphing with Diffusion Models
Text-conditioned image generation models are a prevalent use of AI image synthesis, yet intuitively controlling output guided by an artist remains challenging. Current methods require multiple images and textual prompts …
Image GenerationImage MorphingEnd-to-End Chinese Landscape Painting Creation Using Generative Adversarial Networks
Current GAN-based art generation methods produce unoriginal artwork due to their dependence on conditional input. Here, we propose Sketch-And-Paint GAN (SAPGAN), the first model which generates Chinese landscape painting…
Chinese Landscape Painting GenerationTranslationMagicColor: Multi-Instance Sketch Colorization
We present \textit{MagicColor}, a diffusion-based framework for multi-instance sketch colorization. The production of multi-instance 2D line art colorization adheres to an industry-standard workflow, which consists of th…
ColorizationLine Art ColorizationSketch ColorizationSketching With Style: Visual Search With Sketches and Aesthetic Context
We propose a novel measure of visual similarity for image retrieval that incorporates both structural and aesthetic (style) constraints. Our algorithm accepts a query as sketched shape, and a set of one or more contextua…
Image RetrievalRetrievalTriplet