Calibrating for Class Weights by Modeling Machine Learning
A much studied issue is the extent to which the confidence scores provided by machine learning algorithms are calibrated to ground truth probabilities. Our starting point is that calibration is seemingly incompatible with class weighting, a technique often employed when one class is less common (class imbalance) or with the hope of achieving some external objective (cost-sensitive learning). We provide a model-based explanation for this incompatibility and use our anthropomorphic model to generate a simple method of recovering likelihoods from an algorithm that is miscalibrated due to class weighting. We validate this approach in the binary pneumonia detection task of Rajpurkar, Irvin, Zhu, et al. (2017).
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningPneumonia DetectionSimilar Papers 제목 키워드 기반
Temporally-Adaptive Models for Efficient Video Understanding
Spatial convolutions are extensively used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporal…
Action ClassificationAction RecognitionVideo UnderstandingContext-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model
Despite the success of deep neural network (DNN) on sequential data (i.e., scene text and speech) recognition, it suffers from the over-confidence problem mainly due to overfitting in training with the cross-entropy loss…
Decision MakingScene Text Recognitionspeech-recognitionSpeech RecognitionA neural network-based framework for financial model calibration
A data-driven approach called CaNN (Calibration Neural Network) is proposed to calibrate financial asset price models using an Artificial Neural Network (ANN). Determining optimal values of the model parameters is formul…
BIG-bench Machine Learningglobal-optimizationAdaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport
Few-shot classification aims to learn a classifier to recognize unseen classes during training, where the learned model can easily become over-fitted based on the biased distribution formed by only a few training example…
Domain GeneralizationFew-Shot LearningLearn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) d…
BIG-bench Machine LearningInstance SegmentationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATION+2