Calibrating sufficiently
When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Temporal Probability Calibration
In many applications, accurate class probability estimates are required, but many types of models produce poor quality probability estimates despite achieving acceptable classification accuracy. Even though probability c…
Approximately Optimal Auctions With a Strong Bidder
We consider auctions with N+1 bidders. Of these, N are symmetric and N+1 is "sufficiently strong" relative to the others. The auction is a "tournament" in which the first N players bid to win the right to compete with N+…
Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation
Reliable uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and leverage predictive models. However, existing approaches to estimating uncertainty largely ignore…
Domain AdaptationPredictionUnsupervised Domain AdaptationDeep intra-operative illumination calibration of hyperspectral cameras
Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. Currently available cameras, however, suffer from poor integration into the clinical workflow be…
parameter estimationScene SegmentationUnsupervised Domain Adaptation via Calibrating Uncertainties
Unsupervised domain adaptation (UDA) aims at inferring class labels for unlabeled target domain given a related labeled source dataset. Intuitively, a model trained on source domain normally produces higher uncertainties…
Domain AdaptationUnsupervised Domain Adaptation