Multimodal Conditional Image Synthesis with Product-of-Experts GANs
Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting. The project website is available at https://deepimagination.github.io/PoE-GAN .
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityImage GenerationImage-to-Image TranslationSimilar Papers 제목 키워드 기반
Unified Cross-Modal Image Synthesis with Hierarchical Mixture of Product-of-Experts
We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challeng…
Image GenerationMultimodal Conditionality for Natural Language Generation
Large scale pretrained language models have demonstrated state-of-the-art performance in language understanding tasks. Their application has recently expanded into multimodality learning, leading to improved representati…
DescriptiveLanguage ModelingLanguage ModellingText GenerationMMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
Existing multimodal conditional image synthesis (MCIS) methods generate images conditioned on any combinations of various modalities that require all of them must be exactly conformed, hindering the synthesis controllabi…
Image GenerationMultimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation
Many tasks in computer vision and graphics fall within the framework of conditional image synthesis. In recent years, generative adversarial nets (GANs) have delivered impressive advances in quality of synthesized images…
Image GenerationImage Super-ResolutionSuper-ResolutionPRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities
Missing modalities are prevalent in real-world Alzheimer's disease (AD) assessment and pose a significant challenge to multimodal learning, particularly when the distribution of observed modality subsets differs between …