No Conditional Models for me: Training Joint EBMs on Mixed Continuous and Discrete Data
We propose energy-based models of the joint distribution of data and supervision. While challenging to work with, this approach gives flexibility when designing energy functions and easy parameterization for structured supervision. Further, these models naturally allow training on partially observed data and predictions conditioned on any subset of the modeled variables. We identify and address the main difficulty in working with these models: sampling from the joint distribution of data and supervision. We build upon recent developments in discrete MCMC sampling and apply them alongside continuous MCMC techniques developed for energy-based models. We present experimental results demonstrating that our proposed approach can successfully train joint energy-based models on high-dimensional data with structured supervision capable of both accurate prediction and conditional sampling.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Guiding Energy-based Models via Contrastive Latent Variables
An energy-based model (EBM) is a popular generative framework that offers both explicit density and architectural flexibility, but training them is difficult since it is often unstable and time-consuming. In recent years…
Representation LearningM-EBM: Towards Understanding the Manifolds of Energy-Based Models
Energy-based models (EBMs) exhibit a variety of desirable properties in predictive tasks, such as generality, simplicity and compositionality. However, training EBMs on high-dimensional datasets remains unstable and expe…
Image GenerationEnergy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces
Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on data in discrete or mixed state spaces poses significant challenges due to the lack…
Synthetic Data GenerationDirectly Training Joint Energy-Based Models for Conditional Synthesis and Calibrated Prediction of Multi-Attribute Data
Multi-attribute classification generalizes classification, presenting new challenges for making accurate predictions and quantifying uncertainty. We build upon recent work and show that architectures for multi-attribute …
AttributeEnergy-Based Models with Applications to Speech and Language Processing
Energy-Based Models (EBMs) are an important class of probabilistic models, also known as random fields and undirected graphical models. EBMs are un-normalized and thus radically different from other popular self-normaliz…
Language ModelingLanguage ModellingNatural Language Understandingspeech-recognition+2