Isotropy Maximization Loss and Entropic Score: Accurate, Fast, Efficient, Scalable, and Turnkey Neural Networks Out-of-Distribution Detection Based on The Principle of Maximum Entropy
Current out-of-distribution (OOD) detection approaches require cumbersome procedures that add undesired side effects to the solution. In this paper, we argue that the low OOD detection performance of neural networks is due to cross-entropy SoftMax loss anisotropy and extreme propensity to produce low entropy (high confidence) posterior probability distributions in frontal disagreement with the Principle of Maximum Entropy. Consequently, we propose IsoMax, a loss that is isotropic (distance-based) and produces high entropy (low confidence) posterior probability distributions despite still relying on cross-entropy minimization. Additionally, we propose a speedy Entropic Score for OOD detection. IsoMax loss works as a seamless SoftMax loss drop-in replacement that keeps the overall solution accurate, fast, efficient, scalable, and turnkey. Our experiments indeed confirmed that neural networks OOD detection performance may be extremely improved without relying on techniques such as adversarial training or validation, data augmentation, ensembles methods, generative approaches, model architectural changes, metric learning, or additional classifiers or regressions. The results also showed that our straightforward approach is competitive against state-of-the-art solutions besides avoiding previous methods undesired drawbacks.
Code (1)
Tasks
Data AugmentationMetric LearningOut-of-Distribution DetectionOut of Distribution (OOD) DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhanced Isotropy Maximization Loss: Seamless and High-Performance Out-of-Distribution Detection Simply Replacing the SoftMax Loss
Current out-of-distribution detection approaches usually present special requirements (e.g., collecting outlier data and hyperparameter validation) and produce side effects (e.g., classification accuracy drop and slow/in…
Out-of-Distribution DetectionPartial local entropy and anisotropy in deep weight spaces
We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can …
image-classificationImage ClassificationImproving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score
Current out-of-distribution detection approaches usually present special requirements (e.g., collecting outlier data and hyperparameter validation) and produce side effects (classification accuracy drop and slow/ineffici…
Out-of-Distribution DetectionIsoScore: Measuring the Uniformity of Embedding Space Utilization
The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models …
Word EmbeddingsTsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP pro…
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)