Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias
Variational Autoencoders are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on a lower-dimensional manifold. Recent work by Dai and Wipf (2020) proposes a two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. They gave partial support for that conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics. In this paper, we show that for linear encoders/decoders, the conjecture is true-that is the VAE training does recover a generator with support equal to the ground truth manifold-and does so due to an implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.
Code (1)
Similar Papers 제목 키워드 기반
Interpretable Embeddings From Molecular Simulations Using Gaussian Mixture Variational Autoencoders
Extracting insight from the enormous quantity of data generated from molecular simulations requires the identification of a small number of collective variables whose corresponding low-dimensional free-energy landscape r…
ClusteringDimensionality ReductionPhysical IntuitionVariational Composite Autoencoders
Learning in the latent variable model is challenging in the presence of the complex data structure or the intractable latent variable. Previous variational autoencoders can be low effective due to the straightforward enc…
DecoderPerturbation theory approach to study the latent space degeneracy of Variational Autoencoders
The use of Variational Autoencoders in different Machine Learning tasks has drastically increased in the last years. They have been developed as denoising, clustering and generative tools, highlighting a large potential …
BIG-bench Machine LearningClusteringDenoisingVariational embedding of protein folding simulations using gaussian mixture variational autoencoders
Conformational sampling of biomolecules using molecular dynamics simulations often produces large amount of high dimensional data that makes it difficult to interpret using conventional analysis techniques. Dimensionalit…
Dimensionality ReductionProtein FoldingFONDUE: an algorithm to find the optimal dimensionality of the latent representations of variational autoencoders
When training a variational autoencoder (VAE) on a given dataset, determining the optimal number of latent variables is mostly done by grid search: a costly process in terms of computational time and carbon footprint. In…