VHEGAN: Variational Hetero-Encoder Randomized GAN for Zero-Shot Learning
To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilisitic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network. To further improve VHE and add an image generator, we propose VHE randomized generative adversarial net (VHEGAN) that exploits the synergy between VHE and GAN through their shared latent space. After training with a hybrid stochastic-gradient MCMC/variational inference/stochastic gradient descent inference algorithm, VHEGAN can be used in a variety of settings, such as text generation/retrieval conditioning on an image, image generation/retrieval conditioning on a document/image, and generation of text-image pairs. The efficacy of VHEGAN is demonstrated quantitatively with experiments on both conventional and generalized ZSL tasks, and qualitatively on (conditional) image and/or text generation/retrieval.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationRetrievalText GenerationVariational InferenceZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling
For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, pr…
DecoderGenerative Adversarial NetworkRandomized Benchmarking of Local Zeroth-Order Optimizers for Variational Quantum Systems
In the field of quantum information, classical optimizers play an important role. From experimentalists optimizing their physical devices to theorists exploring variational quantum algorithms, many aspects of quantum inf…
BenchmarkingVariational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data
Learning from heterogeneous data poses challenges such as combining data from various sources and of different types. Meanwhile, heterogeneous data are often associated with missingness in real-world applications due to …
ImputationVariational Graph Normalized Auto-Encoders
Link prediction is one of the key problems for graph-structured data. With the advancement of graph neural networks, graph autoencoders (GAEs) and variational graph autoencoders (VGAEs) have been proposed to learn graph …
Link PredictionPredictionCross-Linked Variational Autoencoders for Generalized Zero-Shot Learning
Most approaches in generalized zero-shot learning rely on cross-modal mapping between an image feature space and a class embedding space or on generating artificial image features. However, learning a shared cross-modal …
Few-Shot LearningGeneralized Zero-Shot LearningZero-Shot Learning