Learning Multimodal Latent Space with EBM Prior and MCMC Inference
Multimodal generative models are crucial for various applications. We propose an approach that combines an expressive energy-based model (EBM) prior with Markov Chain Monte Carlo (MCMC) inference in the latent space for multimodal generation. The EBM prior acts as an informative guide, while MCMC inference, specifically through short-run Langevin dynamics, brings the posterior distribution closer to its true form. This method not only provides an expressive prior to better capture the complexity of multimodality but also improves the learning of shared latent variables for more coherent generation across modalities. Our proposed method is supported by empirical experiments, underscoring the effectiveness of our EBM prior with MCMC inference in enhancing cross-modal and joint generative tasks in multimodal contexts.
Code (0)
등록된 구현이 없습니다.
Tasks
multimodal generationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision
Energy-based models (EBMs) are a flexible class of deep generative models and are well-suited to capture complex dependencies in multimodal data. However, learning multimodal EBM by maximum likelihood requires Markov Cha…
A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference
We study a normalizing flow in the latent space of a top-down generator model, in which the normalizing flow model plays the role of the informative prior model of the generator. We propose to jointly learn the latent sp…
Anomaly DetectionImage GenerationImage InpaintingImage ReconstructionLearning Joint Latent Space EBM Prior Model for Multi-layer Generator
This paper studies the fundamental problem of learning multi-layer generator models. The multi-layer generator model builds multiple layers of latent variables as a prior model on top of the generator, which benefits lea…
Image GenerationOutlier DetectionLearning Energy-Based Prior Model with Diffusion-Amortized MCMC
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in the field of generative modeling due to its flexibility in the formulation and strong modeling power of the late…
validImproving Sampling from Generative Autoencoders with Markov Chains
We focus on generative autoencoders, such as variational or adversarial autoencoders, which jointly learn a generative model alongside an inference model. Generative autoencoders are those which are trained to softly enf…