Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labels
Classification is a fundamental task in many applications on which data-driven methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the optimal performance. This is mainly because the best achievable performance is typically unknown and hence, effectively estimating it is of prime importance. In this paper, we consider binary classification problems and we propose an estimator for the false positive rate (FPR) of the Bayes classifier, that is, the optimal classifier with respect to accuracy, from a given dataset. Our method utilizes soft labels, or real-valued labels, which are gaining significant traction thanks to their properties. We thoroughly examine various theoretical properties of our estimator, including its consistency, unbiasedness, rate of convergence, and variance. To enhance the versatility of our estimator beyond soft labels, we also consider noisy labels, which encompass binary labels. For noisy labels, we develop effective FPR estimators by leveraging a denoising technique and the Nadaraya-Watson estimator. Due to the symmetry of the problem, our results can be readily applied to estimate the false negative rate of the Bayes classifier.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationDenoisingSimilar Papers 제목 키워드 기반
Blending Data and Physics Against False Data Injection Attack: An Event-Triggered Moving Target Defence Approach
Fast and accurate detection of cyberattacks is a key element for a cyber-resilient power system. Recently, data-driven detectors and physics-based Moving Target Defences (MTD) have been proposed to detect false data inje…
State EstimationInfluence of uncertainty estimation techniques on false-positive reduction in liver lesion detection
Deep learning techniques show success in detecting objects in medical images, but still suffer from false-positive predictions that may hinder accurate diagnosis. The estimated uncertainty of the neural network output ha…
Lesion DetectionUsing uncertainty estimation to reduce false positives in liver lesion detection
Despite the successes of deep learning techniques at detecting objects in medical images, false positive detections occur which may hinder an accurate diagnosis. We propose a technique to reduce false positive detections…
Lesion DetectionInformation criteria for structured parameter selection in high dimensional tree and graph models
Parameter selection in high-dimensional models is typically finetuned in a way that keeps the (relative) number of false positives under control. This is because otherwise the few true positives may be dominated by the m…
Multi-Person Pose Estimation Evaluation Using Optimal Transportation and Improved Pose Matching
In Multi-Person Pose Estimation, many metrics place importance on ranking of pose detection confidence scores. Current metrics tend to disregard false-positive poses with low confidence, focusing primarily on a larger nu…
Multi-Person Pose Estimation