An information-theoretic analysis of deep latent-variable models
We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.
Code (0)
등록된 구현이 없습니다.
Tasks
Variational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Asymptotic Accuracy of Distribution-Based Estimation for Latent Variables
Hierarchical statistical models are widely employed in information science and data engineering. The models consist of two types of variables: observable variables that represent the given data and latent variables for t…
Active LearningModel SelectionInformation-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables
Latent variables (LVs) play a crucial role in encoder-decoder models by enabling effective data compression, prediction, and generation. Although their theoretical properties, such as generalization, have been extensivel…
Data CompressionDecoderIdentifying Informative Latent Variables Learned by GIN via Mutual Information
How to learn a good representation of data is one of the most important topics of machine learning. Disentanglement of representations, though believed to be the core feature of good representations, has caused a lot of…
Adversarial AttackDisentanglementOutlier DetectionAsymptotic Accuracy of Bayes Estimation for Latent Variables with Redundancy
Hierarchical parametric models consisting of observable and latent variables are widely used for unsupervised learning tasks. For example, a mixture model is a representative hierarchical model for clustering. From the s…
ClusteringEffects of Additional Data on Bayesian Clustering
Hierarchical probabilistic models, such as mixture models, are used for cluster analysis. These models have two types of variables: observable and latent. In cluster analysis, the latent variable is estimated, and it is …
ClusteringTransfer Learning