Entropic Issues in Likelihood-Based OOD Detection
Deep generative models trained by maximum likelihood remain very popular methods for reasoning about data probabilistically. However, it has been observed that they can assign higher likelihoods to out-of-distribution (OOD) data than in-distribution data, thus calling into question the meaning of these likelihood values. In this work we provide a novel perspective on this phenomenon, decomposing the average likelihood into a KL divergence term and an entropy term. We argue that the latter can explain the curious OOD behaviour mentioned above, suppressing likelihood values on datasets with higher entropy. Although our idea is simple, we have not seen it explored yet in the literature. This analysis provides further explanation for the success of OOD detection methods based on likelihood ratios, as the problematic entropy term cancels out in expectation. Finally, we discuss how this observation relates to recent success in OOD detection with manifold-supported models, for which the above decomposition does not hold directly.
Code (0)
등록된 구현이 없습니다.
Tasks
Out of Distribution (OOD) DetectionSimilar Papers 제목 키워드 기반
A note on the relations between mixture models, maximum-likelihood and entropic optimal transport
This note aims to demonstrate that performing maximum-likelihood estimation for a mixture model is equivalent to minimizing over the parameters an optimal transport problem with entropic regularization. The objective is …
On Model-Based Clustering With Entropic Optimal Transport
We develop a new methodology for model-based clustering. Optimizing the log-likelihood provides a principled statistical framework for clustering, with solutions found via the EM algorithm. However, because the log-likel…
Image SegmentationSub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator
It is well known that, under standard regularity conditions, the maximum likelihood estimator (MLE) satisfies a central limit theorem and converges in distribution to a Gaussian random variable as the sample size grows. …
Generalised Bayesian distance-based phylogenetics for the genomics era
As whole genomes become widely available, maximum likelihood and Bayesian phylogenetic methods are demonstrating their limits in meeting the escalating computational demands. Conversely, distance-based phylogenetic metho…
Bayesian InferenceImproving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score
Current out-of-distribution detection approaches usually present special requirements (e.g., collecting outlier data and hyperparameter validation) and produce side effects (classification accuracy drop and slow/ineffici…
Out-of-Distribution Detection