Higher Order Mutual Information Approximation for Feature Selection
Feature selection is a process of choosing a subset of relevant features so that the quality of prediction models can be improved. An extensive body of work exists on information-theoretic feature selection, based on maximizing Mutual Information (MI) between subsets of features and class labels. The prior methods use a lower order approximation, by treating the joint entropy as a summation of several single variable entropies. This leads to locally optimal selections and misses multi-way feature combinations. We present a higher order MI based approximation technique called Higher Order Feature Selection (HOFS). Instead of producing a single list of features, our method produces a ranked collection of feature subsets that maximizes MI, giving better comprehension (feature ranking) as to which features work best together when selected, due to their underlying interdependent structure. Our experiments demonstrate that the proposed method performs better than existing feature selection approaches while keeping similar running times and computational complexity.
Code (0)
등록된 구현이 없습니다.
Tasks
feature selectionSimilar Papers 제목 키워드 기반
MINERVA: Mutual Information Neural Estimation for Supervised Feature Selection
Existing feature filters rely on statistical pair-wise dependence metrics to model feature-target relationships, but this approach may fail when the target depends on higher-order feature interactions rather than individ…
Representation LearningFeature ImportanceRobust Feature Selection by Mutual Information Distributions
Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirica…
Descriptivefeature selectionIncremental LearningQuantum Feature Selection with Higher-Order Binary Optimization on Trapped-Ion Hardware
We present a quantum feature-selection framework based on a higher-order unconstrained binary optimization (HUBO) formulation that explicitly incorporates multivariate dependencies beyond standard quadratic encodings. In…
Efficient High-Order Interaction-Aware Feature Selection Based on Conditional Mutual Information
This study introduces a novel feature selection approach CMICOT, which is a further evolution of filter methods with sequential forward selection (SFS) whose scoring functions are based on conditional mutual information…
feature selectionUnifying Information-Theoretic and Pair-Counting Clustering Similarity
Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typical…