Improving Diffusion Models for Authentic Virtual Try-on in the Wild
This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment, given a pair of images depicting the person and the garment, respectively. Previous works adapt existing exemplar-based inpainting diffusion models for virtual try-on to improve the naturalness of the generated visuals compared to other methods (e.g., GAN-based), but they fail to preserve the identity of the garments. To overcome this limitation, we propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. Our method, coined IDM-VTON, uses two different modules to encode the semantics of garment image; given the base UNet of the diffusion model, 1) the high-level semantics extracted from a visual encoder are fused to the cross-attention layer, and then 2) the low-level features extracted from parallel UNet are fused to the self-attention layer. In addition, we provide detailed textual prompts for both garment and person images to enhance the authenticity of the generated visuals. Finally, we present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity. Our experimental results show that our method outperforms previous approaches (both diffusion-based and GAN-based) in preserving garment details and generating authentic virtual try-on images, both qualitatively and quantitatively. Furthermore, the proposed customization method demonstrates its effectiveness in a real-world scenario. More visualizations are available in our project page: https://idm-vton.github.io
Code (1)
Tasks
Virtual Try-onMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FitVTON: Fit-aware Virtual Try-On via Body-Garment Size Control
While diffusion-based virtual try-on has achieved impressive visual realism, most methods treat the task as 2D inpainting, prioritizing texture preservation over physical plausibility. Consequently, they often produce pl…
Virtual Try-onA Two-stage Personalized Virtual Try-on Framework with Shape Control and Texture Guidance
The Diffusion model has a strong ability to generate wild images. However, the model can just generate inaccurate images with the guidance of text, which makes it very challenging to directly apply the text-guided genera…
Virtual Try-onBooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training
Image-based virtual try-on is an increasingly popular and important task to generate realistic try-on images of the specific person. Recent methods model virtual try-on as image mask-inpaint task, which requires masking …
Data AugmentationVirtual Try-onPerceptual Quality Assessment of Virtual Reality Videos in the Wild
Investigating how people perceive virtual reality (VR) videos in the wild (i.e., those captured by everyday users) is a crucial and challenging task in VR-related applications due to complex authentic distortions localiz…
Saliency DetectionVideo Quality AssessmentStreet TryOn: Learning In-the-Wild Virtual Try-On from Unpaired Person Images
Most virtual try-on research is motivated to serve the fashion business by generating images to demonstrate garments on studio models at a lower cost. However, virtual try-on should be a broader application that also all…
Image GenerationSemantic SegmentationVirtual Try-onVirtual Try-on (Model2Street)+2