Learning in Variational Autoencoders with Kullback-Leibler and Renyi Integral Bounds
In this paper we propose two novel bounds for the log-likelihood based on Kullback-Leibler and the R\'{e}nyi divergences, which can be used for variational inference and in particular for the training of Variational AutoEncoders. Our proposal is motivated by the difficulties encountered in training VAEs on continuous datasets with high contrast images, such as those with handwritten digits and characters, where numerical issues often appear unless noise is added, either to the dataset during training or to the generative model given by the decoder. The new bounds we propose, which are obtained from the maximization of the likelihood of an interval for the observations, allow numerically stable training procedures without the necessity of adding any extra source of noise to the data.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderVariational InferenceSimilar Papers 제목 키워드 기반
Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function
In Bayesian machine learning, the posterior distribution is typically computationally intractable, hence variational inference is often required. In this approach, an evidence lower bound on the log likelihood of data is…
Variational InferenceOn the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation
Variational Autoencoders (VAEs) are known to suffer from learning uninformative latent representation of the input due to issues such as approximated posterior collapse, or entanglement of the latent space. We impose an …
Text GenerationBalancing reconstruction error and Kullback-Leibler divergence in Variational Autoencoders
In the loss function of Variational Autoencoders there is a well known tension between two components: the reconstruction loss, improving the quality of the resulting images, and the Kullback-Leibler divergence, acting a…
Mixture weights optimisation for Alpha-Divergence Variational Inference
This paper focuses on $\alpha$-divergence minimisation methods for Variational Inference. More precisely, we are interested in algorithms optimising the mixture weights of any given mixture model, without any information…
Variational InferenceMutual Information Constraints for Monte-Carlo Objectives
A common failure mode of density models trained as variational autoencoders is to model the data without relying on their latent variables, rendering these variables useless. Two contributing factors, the underspecificat…