Learning Discrete Structured Representations by Adversarially Maximizing Mutual Information
We propose learning discrete structured representations from unlabeled data by maximizing the mutual information between a structured latent variable and a target variable. Calculating mutual information is intractable in this setting. Our key technical contribution is an adversarial objective that can be used to tractably estimate mutual information assuming only the feasibility of cross entropy calculation. We develop a concrete realization of this general formulation with Markov distributions over binary encodings. We report critical and unexpected findings on practical aspects of the objective such as the choice of variational priors. We apply our model on document hashing and show that it outperforms current best baselines based on discrete and vector quantized variational autoencoders. It also yields highly compressed interpretable representations.
Code (1)
Similar Papers 제목 키워드 기반
Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization
Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. To better understand adversarial robustness, we consider the underlying problem of learning robust re…
Adversarial RobustnessLearning deep representations by mutual information estimation and maximization
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporat…
General ClassificationMutual Information EstimationRepresentation LearningINFORMATION MAXIMIZATION AUTO-ENCODING
We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting. Unlike the Variational Autoencoder…
DecoderDisentanglementInformativenessvalidMultivariate-Information Adversarial Ensemble for Scalable Joint Distribution Matching
A broad range of cross-$m$-domain generation researches boil down to matching a joint distribution by deep generative models (DGMs). Hitherto algorithms excel in pairwise domains while as $m$ increases, remain struggling…
Deep Graph Infomax
We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual information between patch representations …
General ClassificationInductive LearningNode Classification