Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax
Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM.
Code (0)
등록된 구현이 없습니다.
Tasks
DisentanglementRepresentation LearningSimilar Papers 제목 키워드 기반
Smooth InfoMax -- Towards easier Post-Hoc interpretability
We introduce Smooth InfoMax (SIM), a novel method for self-supervised representation learning that incorporates an interpretability constraint into the learned representations at various depths of the neural network. SIM…
DecoderRepresentation LearningHTCInfoMax: A Global Model for Hierarchical Text Classification via Information Maximization
The current state-of-the-art model HiAGM for hierarchical text classification has two limitations. First, it correlates each text sample with all labels in the dataset which contains irrelevant information. Second, it do…
General ClassificationRepresentation Learningtext-classificationText ClassificationFlow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline…
Reinforcement LearningOffline RLLearning deep representations by mutual information estimation and maximization
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporat…
General ClassificationMutual Information EstimationRepresentation LearningGraph Embedding Using Infomax for ASD Classification and Brain Functional Difference Detection
Significant progress has been made using fMRI to characterize the brain changes that occur in ASD, a complex neuro-developmental disorder. However, due to the high dimensionality and low signal-to-noise ratio of fMRI, em…
ClassificationGeneral ClassificationGraph EmbeddingGraph Neural Network