Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence
Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence (even under linear transformations designed to enhance the extraction of information), prioritizes redundancy over informative content, and in some cases, performs worse than simpler dependence measures like the correlation coefficient.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingSimilar Papers 제목 키워드 기반
Slicing Mutual Information Generalization Bounds for Neural Networks
The ability of machine learning (ML) algorithms to generalize well to unseen data has been studied through the lens of information theory, by bounding the generalization error with the input-output mutual information (MI…
Generalization BoundsModel CompressionSliced Wasserstein with Random-Path Projecting Directions
Slicing distribution selection has been used as an effective technique to improve the performance of parameter estimators based on minimizing sliced Wasserstein distance in applications. Previous works either utilize exp…
DenoisingRevisiting Sliced Wasserstein on Images: From Vectorization to Convolution
The conventional sliced Wasserstein is defined between two probability measures that have realizations as vectors. When comparing two probability measures over images, practitioners first need to vectorize images and the…
Slicing Unbalanced Optimal Transport
Optimal transport (OT) is a powerful framework to compare probability measures, a fundamental task in many statistical and machine learning problems. Substantial advances have been made in designing OT variants which are…
Computational EfficiencyGPUConstrained Sliced Wasserstein Embedding
Sliced Wasserstein (SW) distances offer an efficient method for comparing high-dimensional probability measures by projecting them onto multiple 1-dimensional probability distributions. However, identifying informative s…