Formal Limitations on the Measurement of Mutual Information
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).
Code (2)
Similar Papers 제목 키워드 기반
Mutual Wasserstein Discrepancy Minimization for Sequential Recommendation
Self-supervised sequential recommendation significantly improves recommendation performance by maximizing mutual information with well-designed data augmentations. However, the mutual information estimation is based on t…
Contrastive LearningMutual Information EstimationSelf-Supervised LearningSequential RecommendationInformation-Theoretic Optimization for Task-Adapted Compressed Sensing Magnetic Resonance Imaging
Task-adapted compressed sensing magnetic resonance imaging (CS-MRI) is emerging to address the specific demands of downstream clinical tasks with significantly fewer k-space measurements than required by Nyquist sampling…
Medical DiagnosisWhich Mutual-Information Representation Learning Objectives are Sufficient for Control?
Mutual information maximization provides an appealing formalism for learning representations of data. In the context of reinforcement learning (RL), such representations can accelerate learning by discarding irrelevant a…
Reinforcement Learning (RL)Representation LearningApproximate Computational Approaches for Bayesian Sensor Placement in High Dimensions
Since the cost of installing and maintaining sensors is usually high, sensor locations are always strategically selected. For those aiming at inferring certain quantities of interest (QoI), it is desirable to explore the…
Bayesian OptimizationVocal Bursts Intensity PredictionDesigned Measurements for Vector Count Data
We consider design of linear projection measurements for a vector Poisson signal model. The projections are performed on the vector Poisson rate, $X\in\mathbb{R}_+^n$, and the observed data are a vector of counts, $Y\in\…
Compressive Sensing