Fast and close Shannon entropy approximation
Shannon entropy (SE) and its quantum mechanical analogue von Neumann entropy are key components in many tools used in physics, information theory, machine learning (ML) and quantum computing. Besides of the significant amounts of SE computations required in these fields, the singularity of the SE gradient is one of the central mathematical reason inducing the high cost, frequently low robustness and slow convergence of such tools. Here we propose the Fast Entropy Approximation (FEA) - a non-singular rational approximation of Shannon entropy and its gradient that achieves a mean absolute error of $10^{-3}$, which is approximately $20$ times lower than comparable state-of-the-art methods. FEA allows around $50\%$ faster computation, requiring only $5$ to $6$ elementary computational operations, as compared to tens of elementary operations behind the fastest entropy computation algorithms with table look-ups, bitshifts, or series approximations. On a set of common benchmarks for the feature selection problem in machine learning, we show that the combined effect of fewer elementary operations, low approximation error, and a non-singular gradient allows significantly better model quality and enables ML feature extraction that is two to three orders of magnitude faster and computationally cheaper when incorporating FEA into AI tools.
Code (0)
등록된 구현이 없습니다.
Tasks
feature selectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning
Maximum Tsallis entropy (MTE) framework in reinforcement learning has gained popularity recently by virtue of its flexible modeling choices including the widely used Shannon entropy and sparse entropy. However, non-Shann…
reinforcement-learningReinforcement Learning (RL)A Theory on AI Uncertainty Based on Rademacher Complexity and Shannon Entropy
In this paper, we present a theoretical discussion on AI deep learning neural network uncertainty investigation based on the classical Rademacher complexity and Shannon entropy. First it is shown that the classical Radem…
General ClassificationVariance of entropy for testing time-varying regimes with an application to meme stocks
Shannon entropy is the most common metric to measure the degree of randomness of time series in many fields, ranging from physics and finance to medicine and biology. Real-world systems may be in general non stationary, …
Time SeriesTime Series AnalysisAI Uncertainty Based on Rademacher Complexity and Shannon Entropy
In this paper from communication channel coding perspective we are able to present both a theoretical and practical discussion of AI's uncertainty, capacity and evolution for pattern classification based on the classical…
ClassificationGeneral ClassificationA Bayesian Monte-Carlo Uncertainty Model for Assessment of Shear Stress Entropy
The entropy models have been recently adopted in many studies to evaluate the distribution of the shear stress in circular channels. However, the uncertainty in their predictions and their reliability remains an open que…
Open-Ended Question Answering