How do Variational Autoencoders Learn? Insights from Representational Similarity
The ability of Variational Autoencoders (VAEs) to learn disentangled representations has made them popular for practical applications. However, their behaviour is not yet fully understood. For example, the questions of when they can provide disentangled representations, or suffer from posterior collapse are still areas of active research. Despite this, there are no layerwise comparisons of the representations learned by VAEs, which would further our understanding of these models. In this paper, we thus look into the internal behaviour of VAEs using representational similarity techniques. Specifically, using the CKA and Procrustes similarities, we found that the encoders' representations are learned long before the decoders', and this behaviour is independent of hyperparameters, learning objectives, and datasets. Moreover, the encoders' representations in all but the mean and variance layers are similar across hyperparameters and learning objectives.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How good are variational autoencoders at transfer learning?
Variational autoencoders (VAEs) are used for transfer learning across various research domains such as music generation or medical image analysis. However, there is no principled way to assess before transfer which compo…
Medical Image AnalysisMusic GenerationTransfer LearningTheoretical Insights into the Use of Structural Similarity Index In Generative Models and Inferential Autoencoders
Generative models and inferential autoencoders mostly make use of $\ell_2$ norm in their optimization objectives. In order to generate perceptually better images, this short paper theoretically discusses how to use Struc…
Dimensionality ReductionImage GenerationImage Quality AssessmentSSIMPosterior Collapse as a Phase Transition in Variational Autoencoders
We investigate the phenomenon of posterior collapse in variational autoencoders (VAEs) from the perspective of statistical physics, and reveal that it constitutes a phase transition governed jointly by data structure and…
Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple…
ClusteringRepresentation LearningVariational InferenceThe Neglected Sibling: Isotropic Gaussian Posterior for VAE
Deep generative models have been widely used in several areas of NLP, and various techniques have been proposed to augment them or address their training challenges. In this paper, we propose a simple modification to Var…