Neural Network Classifier as Mutual Information Evaluator
Cross-entropy loss with softmax output is a standard choice to train neural network classifiers. We give a new view of neural network classifiers with softmax and cross-entropy as mutual information evaluators. We show that when the dataset is balanced, training a neural network with cross-entropy maximises the mutual information between inputs and labels through a variational form of mutual information. Thereby, we develop a new form of softmax that also converts a classifier to a mutual information evaluator when the dataset is imbalanced. Experimental results show that the new form leads to better classification accuracy, in particular for imbalanced datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
FormMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems
The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balanc…
PositionMaximum mutual information regularized classification
In this paper, a novel pattern classification approach is proposed by regularizing the classifier learning to maximize mutual information between the classification response and the true class label. We argue that, with …
ClassificationGeneral ClassificationMutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems
Deep learning systems have been reported to acheive state-of-the-art performances in many applications, and one of the keys for achieving this is the existence of well trained classifiers on benchmark datasets which can …
Binary ClassificationData AugmentationMutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems
Deep learning systems have been reported to achieve state-of-the-art performances in many applications, and a key is the existence of well trained classifiers on benchmark datasets. As a main-stream loss function, the cr…
Binary ClassificationDerivations of Normalized Mutual Information in Binary Classifications
This correspondence studies the basic problem of classifications - how to evaluate different classifiers. Although the conventional performance indexes, such as accuracy, are commonly used in classifier selection or eval…
Form