Variational Information Maximization for Feature Selection
Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the difficulty of estimating mutual information. We demonstrate that approximations made by existing methods are based on unrealistic assumptions. We formulate a more flexible and general class of assumptions based on variational distributions and use them to tractably generate lower bounds for mutual information. These bounds define a novel information-theoretic framework for feature selection, which we prove to be optimal under tree graphical models with proper choice of variational distributions. Our experiments demonstrate that the proposed method strongly outperforms existing information-theoretic feature selection approaches.
Code (1)
Tasks
feature selectionSimilar Papers 제목 키워드 기반
High-Order Conditional Mutual Information Maximization for dealing with High-Order Dependencies in Feature Selection
This paper presents a novel feature selection method based on the conditional mutual information (CMI). The proposed High Order Conditional Mutual Information Maximization (HOCMIM) incorporates high order dependencies in…
feature selectionVocal Bursts Intensity PredictionAPS: Active Pretraining with Successor Features
We introduce a new unsupervised pretraining objective for reinforcement learning. During the unsupervised reward-free pretraining phase, the agent maximizes mutual information between tasks and states induced by the poli…
Unsupervised Reinforcement LearningA Variational EM Framework With Adaptive Edge Selection for Blind Motion Deblurring
Blind motion deblurring is an important problem that receives enduring attention in last decade. Based on the observation that a good intermediate estimate of latent image for estimating motion-blur kernel is not neces…
DeblurringA Variational Approximations-DIC Rubric for Parameter Estimation and Mixture Model Selection Within a Family Setting
Mixture model-based clustering has become an increasingly popular data analysis technique since its introduction over fifty years ago, and is now commonly utilized within a family setting. Families of mixture models aris…
ClusteringModel Selectionparameter estimationAn Efficient Model Selection for Gaussian Mixture Model in a Bayesian Framework
In order to cluster or partition data, we often use Expectation-and-Maximization (EM) or Variational approximation with a Gaussian Mixture Model (GMM), which is a parametric probability density function represented as a …
ClusteringmodelModel Selection