Shape-Guided Diffusion with Inside-Outside Attention
We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Effective Video Mirror Detection with Inconsistent Motion Cues
Image-based mirror detection has recently undergone rapid research due to its significance in applications such as robotic navigation semantic segmentation and scene reconstruction. Recently VMD-Net was proposed as t…
Edge DetectionMirror DetectionOptical Flow EstimationSemantic SegmentationOctree Guided Unoriented Surface Reconstruction
We address the problem of surface reconstruction from unoriented point clouds. Implicit neural representations (INRs) have become popular for this task, but when information relating to the inside versus outside of a…
Surface ReconstructionOSSO: Obtaining Skeletal Shape from Outside
We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i.e. we predict the inside (bones) from the outside (skin). This has many applications in medi…
GIFS: Neural Implicit Function for General Shape Representation
Recent development of neural implicit function has shown tremendous success on high-quality 3D shape reconstruction. However, most works divide the space into inside and outside of the shape, which limits their represent…
3D Shape ReconstructionPhysics-guided training of GAN to improve accuracy in airfoil design synthesis
Generative adversarial networks (GAN) have recently been used for a design synthesis of mechanical shapes. A GAN sometimes outputs physically unreasonable shapes. For example, when a GAN model is trained to output airfoi…
Design Synthesis