Variational Distribution Learning for Unsupervised Text-to-Image Generation

2023-03-28 · CVPR 2023 1 · Minsoo Kang, Doyup Lee, Jiseob Kim, Saehoon Kim, Bohyung Han

We propose a text-to-image generation algorithm based on deep neural networks when text captions for images are unavailable during training. In this work, instead of simply generating pseudo-ground-truth sentences of training images using existing image captioning methods, we employ a pretrained CLIP model, which is capable of properly aligning embeddings of images and corresponding texts in a joint space and, consequently, works well on zero-shot recognition tasks. We optimize a text-to-image generation model by maximizing the data log-likelihood conditioned on pairs of image-text CLIP embeddings. To better align data in the two domains, we employ a principled way based on a variational inference, which efficiently estimates an approximate posterior of the hidden text embedding given an image and its CLIP feature. Experimental results validate that the proposed framework outperforms existing approaches by large margins under unsupervised and semi-supervised text-to-image generation settings.

📄 PDF Abstract BibTeX arXiv:2303.16105

Code (0)

등록된 구현이 없습니다.

Tasks

Image Captioning Image Generation Text to Image Generation Text-to-Image GenerationVariational InferenceZero-Shot Learning

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Multi-Adversarial Variational Autoencoder Networks

2019-06-14 · Abdullah-Al-Zubaer Imran, Demetri Terzopoulos

The unsupervised training of GANs and VAEs has enabled them to generate realistic images mimicking real-world distributions and perform image-based unsupervised clustering or semi-supervised classification. Combining the…

Clustering General Classification Image Generation Variational Inference

Learning Disentangled Discrete Representations

2023-07-26 · David Friede, Christian Reimers, Heiner Stuckenschmidt, Mathias Niepert

Recent successes in image generation, model-based reinforcement learning, and text-to-image generation have demonstrated the empirical advantages of discrete latent representations, although the reasons behind their bene…

Image Generation Model-based Reinforcement Learning Model Selection Text to Image Generation+1

VHEGAN: Variational Hetero-Encoder Randomized GAN for Zero-Shot Learning

2019-05-01 · ICLR 2019 5 · Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang 외

To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilisitic t…

Image Generation Retrieval Text Generation Variational Inference+1

Multiresolution Equivariant Graph Variational Autoencoder

2021-06-02 · Truong Son Hy, Risi Kondor

In this paper, we propose Multiresolution Equivariant Graph Variational Autoencoders (MGVAE), the first hierarchical generative model to learn and generate graphs in a multiresolution and equivariant manner. At each reso…

Graph Generation Image Generation Link Prediction molecular representation+1

DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping

2024-09-08 · Zeyu Cai, Duotun Wang, Yixun Liang, Zhijing Shao 외

Score Distillation Sampling (SDS) has emerged as a prevalent technique for text-to-3D generation, enabling 3D content creation by distilling view-dependent information from text-to-2D guidance. However, they frequently e…

3D Generation Text to 3D