Deep Variational Information Bottleneck
We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.
Code (9)
Tasks
Adversarial AttackSimilar Papers 제목 키워드 기반
Variational Information Bottleneck on Vector Quantized Autoencoders
In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational determ…
Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses
Variational dimensionality reduction methods are widely used for their accuracy, generative capabilities, and robustness. We introduce a unifying framework that generalizes both such as traditional and state-of-the-art m…
Contrastive LearningDecoderDimensionality ReductionRepresentation LearningThe Variational Deficiency Bottleneck
We introduce a bottleneck method for learning data representations based on information deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method e…
General ClassificationVariational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding
In this paper, we develop an unsupervised generative clustering framework that combines the Variational Information Bottleneck and the Gaussian Mixture Model. Specifically, in our approach, we use the Variational Informa…
ClusteringVariational InferenceVariational Predictive Information Bottleneck
In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this functional as a variational lower bound …
Bayesian Inference