Mutual Information Neural Estimation
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement the Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.
Code (1)
Tasks
General ClassificationMutual Information EstimationSimilar Papers 제목 키워드 기반
NMINE: Normalized Mutual Information Neural Estimation
Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensio…
Interpretable Machine LearningExponential Concentration for Mutual Information Estimation with Application to Forests
We prove a new exponential concentration inequality for a plug-in estimator of the Shannon mutual information. Previous results on mutual information estimation only bounded expected error. The advantage of having the ex…
Mutual Information EstimationMatching Text with Deep Mutual Information Estimation
Text matching is a core natural language processing research problem. How to retain sufficient information on both content and structure information is one important challenge. In this paper, we present a neural approach…
Answer SelectionMutual Information EstimationNatural Language InferenceParaphrase Identification+1Discrete Bridges for Mutual Information Estimation
Diffusion bridge models in both continuous and discrete state spaces have recently become powerful tools in the field of generative modeling. In this work, we leverage the discrete state space formulation of bridge match…
On the Properties and Estimation of Pointwise Mutual Information Profiles
The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precise…
Mutual Information EstimationUncertainty Quantification