HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulation
Generative Adversarial Networks (GANs), particularly StyleGAN and its variants, have demonstrated remarkable capabilities in generating highly realistic images. Despite their success, adapting these models to diverse tasks such as domain adaptation, reference-guided synthesis, and text-guided manipulation with limited training data remains challenging. Towards this end, in this study, we present a novel framework that significantly extends the capabilities of a pre-trained StyleGAN by integrating CLIP space via hypernetworks. This integration allows dynamic adaptation of StyleGAN to new domains defined by reference images or textual descriptions. Additionally, we introduce a CLIP-guided discriminator that enhances the alignment between generated images and target domains, ensuring superior image quality. Our approach demonstrates unprecedented flexibility, enabling text-guided image manipulation without the need for text-specific training data and facilitating seamless style transfer. Comprehensive qualitative and quantitative evaluations confirm the robustness and superior performance of our framework compared to existing methods.
Code (1)
Tasks
Domain AdaptationImage GenerationImage ManipulationStyle TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
HyperGAN: Exploring the Manifold of Neural Networks
We introduce HyperGAN, a generative network that learns to generate all the weight parameters of deep neural networks. HyperGAN first transforms low dimensional noise into a latent space, which can be sampled from to obt…
HyperGAN: A Generative Model for Diverse, Performant Neural Networks
Standard neural networks are often overconfident when presented with data outside the training distribution. We introduce HyperGAN, a new generative model for learning a distribution of neural network parameters. HyperGA…
General ClassificationUniHDA: A Unified and Versatile Framework for Multi-Modal Hybrid Domain Adaptation
Recently, generative domain adaptation has achieved remarkable progress, enabling us to adapt a pre-trained generator to a new target domain. However, existing methods simply adapt the generator to a single target domain…
AttributeDiversityDomain AdaptationSplit to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation
Large vision-language models (VLMs) like CLIP have demonstrated good zero-shot learning performance in the unsupervised domain adaptation task. Yet, most transfer approaches for VLMs focus on either the language or visua…
Domain AdaptationUnsupervised Domain AdaptationZero-Shot LearningTest-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation
The Segment Anything Model (SAM) and CLIP are remarkable vision foundation models (VFMs). SAM, a prompt driven segmentation model, excels in segmentation tasks across diverse domains, while CLIP is renowned for its zero …
Image SegmentationMedical Image SegmentationOrgan SegmentationPrompt Engineering+5