Selective Probabilistic Classifier Based on Hypothesis Testing
In this paper, we propose a simple yet effective method to deal with the violation of the Closed-World Assumption for a classifier. Previous works tend to apply a threshold either on the classification scores or the loss function to reject the inputs that violate the assumption. However, these methods cannot achieve the low False Positive Ratio (FPR) required in safety applications. The proposed method is a rejection option based on hypothesis testing with probabilistic networks. With probabilistic networks, it is possible to estimate the distribution of outcomes instead of a single output. By utilizing Z-test over the mean and standard deviation for each class, the proposed method can estimate the statistical significance of the network certainty and reject uncertain outputs. The proposed method was experimented on with different configurations of the COCO and CIFAR datasets. The performance of the proposed method is compared with the Softmax Response, which is a known top-performing method. It is shown that the proposed method can achieve a broader range of operation and cover a lower FPR than the alternative.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Conjunction Subspaces Test for Conformal and Selective Classification
In this paper, we present a new classifier, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision. The null h…
ClassificationConformal PredictionOn the Role of Randomization in Adversarially Robust Classification
Deep neural networks are known to be vulnerable to small adversarial perturbations in test data. To defend against adversarial attacks, probabilistic classifiers have been proposed as an alternative to deterministic ones…
Robust classificationA Statistical Test for Probabilistic Fairness
Algorithms are now routinely used to make consequential decisions that affect human lives. Examples include college admissions, medical interventions or law enforcement. While algorithms empower us to harness all informa…
BIG-bench Machine LearningFairnessI-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers
As probabilistic models continue to permeate various facets of our society and contribute to scientific advancements, it becomes a necessity to go beyond traditional metrics such as predictive accuracy and error rates an…
DiagnosticAsk for More Than Bayes Optimal: A Theory of Indecisions for Classification
Selective classification is a powerful tool for automated decision-making in high-risk scenarios, allowing classifiers to make only highly confident decisions while abstaining when uncertainty is too high. Given a target…
Decision Making