A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features
In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability necessary for efficient stochastic gradient variational inference. To learn the parameters of the new model, we approximate the posterior of the latent variables with a variational auto-encoder. Rather than making the usual mean-field assumption however, the encoder parameterizes a new type of structured variational approximation that retains the prior dependencies of the generative model. Using this structured posterior approximation, we are able to perform joint training of deep models with many layers of latent random variables, without having to resort to stacking or other layerwise training procedures.
Code (0)
등록된 구현이 없습니다.
Tasks
Variational InferenceVocal Bursts Type PredictionSimilar Papers 제목 키워드 기반
Recursive Neural Programs: Variational Learning of Image Grammars and Part-Whole Hierarchies
Human vision involves parsing and representing objects and scenes using structured representations based on part-whole hierarchies. Computer vision and machine learning researchers have recently sought to emulate this ca…
Transfer LearningNonparametric Variational Auto-encoders for Hierarchical Representation Learning
The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple…
ClusteringRepresentation LearningVariational InferenceSparse Gaussian Process Variational Autoencoders
Large, multi-dimensional spatio-temporal datasets are omnipresent in modern science and engineering. An effective framework for handling such data are Gaussian process deep generative models (GP-DGMs), which employ GP pr…
Computational EfficiencyVariational InferenceLearning Deep-Latent Hierarchies by Stacking Wasserstein Autoencoders
Probabilistic models with hierarchical-latent-variable structures provide state-of-the-art results amongst non-autoregressive, unsupervised density-based models. However, the most common approach to training such models …
From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
Sparse autoencoders (SAEs) have proven effective for extracting monosemantic features from large language models (LLMs), yet these features are typically identified in isolation. However, broad evidence suggests that LLM…