Partitioning the Sample Space for a More Precise Shannon Entropy Estimation
Reliable data-driven estimation of Shannon entropy from small data sets, where the number of examples is potentially smaller than the number of possible outcomes, is a critical matter in several applications. In this paper, we introduce a discrete entropy estimator, where we use the decomposability property in combination with estimations of the missing mass and the number of unseen outcomes to compensate for the negative bias induced by them. Experimental results show that the proposed method outperforms some classical estimators in undersampled regimes, and performs comparably with some well-established state-of-the-art estimators.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The CM Algorithm for the Maximum Mutual Information Classifications of Unseen Instances
The Maximum Mutual Information (MMI) criterion is different from the Least Error Rate (LER) criterion. It can reduce failing to report small probability events. This paper introduces the Channels Matching (CM) algorithm …
The Representation Jensen-Shannon Divergence
Quantifying the difference between probability distributions is crucial in machine learning. However, estimating statistical divergences from empirical samples is challenging due to unknown underlying distributions. This…
Domain AdaptationTwo-sample testingUnsupervised Domain AdaptationK-Nearest Neighbors Hashing
Hashing based approximate nearest neighbor search embeds high dimensional data to compact binary codes, which enables efficient similarity search and storage. However, the non-isometry sign() function makes it hard to pr…
Shannon-like Interpolation with Spectral Priors and Weighted Hilbert Spaces: Beyond the Nyquist Rate
In this work, we draw connections between the classical Shannon interpolation of bandlimited deterministic signals and the literature on estimating continuous-time random processes from their samples (known in various co…
Hilbert Exclusion: Improved Metric Search through Finite Isometric Embeddings
Most research into similarity search in metric spaces relies upon the triangle inequality property. This property allows the space to be arranged according to relative distances to avoid searching some subspaces. We show…