Learning Multimodal Latent Generative Models with Energy-Based Prior
Multimodal generative models have recently gained significant attention for their ability to learn representations across various modalities, enhancing joint and cross-generation coherence. However, most existing works use standard Gaussian or Laplacian distributions as priors, which may struggle to capture the diverse information inherent in multiple data types due to their unimodal and less informative nature. Energy-based models (EBMs), known for their expressiveness and flexibility across various tasks, have yet to be thoroughly explored in the context of multimodal generative models. In this paper, we propose a novel framework that integrates the multimodal latent generative model with the EBM. Both models can be trained jointly through a variational scheme. This approach results in a more expressive and informative prior, better-capturing of information across multiple modalities. Our experiments validate the proposed model, demonstrating its superior generation coherence.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Multimodal Latent Space with EBM Prior and MCMC Inference
Multimodal generative models are crucial for various applications. We propose an approach that combines an expressive energy-based model (EBM) prior with Markov Chain Monte Carlo (MCMC) inference in the latent space for …
multimodal generationAn Energy-Based Prior for Generative Saliency
We propose a novel generative saliency prediction framework that adopts an informative energy-based model as a prior distribution. The energy-based prior model is defined on the latent space of a saliency generator netwo…
object-detectionObject DetectionRGB-D Salient Object DetectionSaliency Prediction+2Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction
Vision transformer networks have shown superiority in many computer vision tasks. In this paper, we take a step further by proposing a novel generative vision transformer with latent variables following an informative en…
object-detectionObject DetectionRGB-D Salient Object DetectionSaliency Prediction+2Improving VAE generations of multimodal data through data-dependent conditional priors
One of the major shortcomings of variational autoencoders is the inability to produce generations from the individual modalities of data originating from mixture distributions. This is primarily due to the use of a simpl…
Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision
Energy-based models (EBMs) are a flexible class of deep generative models and are well-suited to capture complex dependencies in multimodal data. However, learning multimodal EBM by maximum likelihood requires Markov Cha…