Learning deep representations by mutual information estimation and maximization
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.
Code (8)
Tasks
General ClassificationMutual Information EstimationRepresentation LearningSimilar Papers 제목 키워드 기반
Multimodal Representation Learning via Maximization of Local Mutual Information
We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking a…
image-classificationImage ClassificationMutual Information EstimationRepresentation LearningLearning Disentangled Representations via Mutual Information Estimation
In this paper, we investigate the problem of learning disentangled representations. Given a pair of images sharing some attributes, we aim to create a low-dimensional representation which is split into two parts: a share…
DisentanglementGeneral Classificationimage-classificationImage Classification+5Multimodal Representations Learning Based on Mutual Information Maximization and Minimization and Identity Embedding for Multimodal Sentiment Analysis
Multimodal sentiment analysis (MSA) is a fundamental complex research problem due to the heterogeneity gap between different modalities and the ambiguity of human emotional expression. Although there have been many succe…
Multimodal Sentiment AnalysisSentiment AnalysisVMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors
Variational Autoencoder is a scalable method for learning latent variable models of complex data. It employs a clear objective that can be easily optimized. However, it does not explicitly measure the quality of learned …
Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representations
Recently, maximizing mutual information has emerged as a powerful method for unsupervised graph representation learning. The existing methods are typically effective to capture information from the topology view but igno…
DiversityGraph Representation LearningLinear evaluationRepresentation Learning