Unsupervised Calibration under Covariate Shift
A probabilistic model is said to be calibrated if its predicted probabilities match the corresponding empirical frequencies. Calibration is important for uncertainty quantification and decision making in safety-critical applications. While calibration of classifiers has been widely studied, we find that calibration is brittle and can be easily lost under minimal covariate shifts. Existing techniques, including domain adaptation ones, primarily focus on prediction accuracy and do not guarantee calibration neither in theory nor in practice. In this work, we formally introduce the problem of calibration under domain shift, and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingDomain AdaptationUncertainty QuantificationSimilar Papers 제목 키워드 기반
Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift
Confidence calibration for classification models is vital in safety-critical decision-making scenarios and has received extensive attention. General confidence calibration methods assume training and test data are indepe…
Unsupervised Domain AdaptationContrastive Conditional Alignment based on Label Shift Calibration for Imbalanced Domain Adaptation
Many existing unsupervised domain adaptation (UDA) methods primarily focus on covariate shift, limiting their effectiveness in imbalanced domain adaptation (IDA) where both covariate shift and label shift coexist. Recent…
Domain AdaptationUnsupervised Domain AdaptationConfidence Calibration for Domain Generalization under Covariate Shift
Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation. However, these methods suffer from the following limitations: 1) they require unlabeled data from the target doma…
Domain AdaptationDomain GeneralizationMulti-class ClassificationUnsupervised Domain AdaptationBridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift
We establish a new model-agnostic optimization framework for out-of-distribution generalization via multicalibration, a criterion that ensures a predictor is calibrated across a family of overlapping groups. Multicalibra…
Out-of-Distribution GeneralizationSimulator Calibration under Covariate Shift with Kernels
We propose a novel calibration method for computer simulators, dealing with the problem of covariate shift. Covariate shift is the situation where input distributions for training and test are different, and ubiquitous i…
Bayesian Inference