Anytime-Valid Confirmation of Label-Shift Corrections
In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation even when unlabeled target inputs are available. We address the complementary setting where the practitioner has a pre-specified label-shift correction from domain knowledge and asks whether incoming labeled outcomes support it. We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value, so its running product is a nonnegative martingale and Ville's inequality yields an anytime-valid confirmation rule. The log martingale equals the cumulative negative log-predictive density (NLPD) gap between the source and the corrected predictive, converting routine model monitoring into a formal sequential test. Rejection means the incoming data support the posited correction relative to the source predictive, but it is not a precise estimate of the degree of shift. Closed forms are available for GP sources with Gaussian label-shift ratios. GP regression simulations validate Type I control, finite-sample power, miscalibration sensitivity, and the small-batch advantage of a reliable prior over label-based re-estimation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Anytime-Valid Confirmation of Covariate Balance for Prespecified Corrections
Many covariate-shift adaptation methods construct a correction $w(x)$, but users must still determine whether the corrected distribution is sufficiently balanced for the target stream. We study anytime-valid confirmation…
Two Wrongs Don't Make a Right: Combating Confirmation Bias in Learning with Label Noise
Noisy labels damage the performance of deep networks. For robust learning, a prominent two-stage pipeline alternates between eliminating possible incorrect labels and semi-supervised training. However, discarding part of…
Image ClassificationDynamic Vehicle Routing Problem with Prompt Confirmation of Advance Requests
Transit agencies that operate on-demand transportation services have to respond to trip requests from passengers in real time, which involves solving dynamic vehicle routing problems with pick-up and drop-off constraints…
Reinforcement LearningAnytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift
A unified framework for learning under covariate shift is presented, in which a constrained density-ratio network approximates the Radon-Nikodym derivative $r^\star = dP/dQ$ and feeds an anytime PAC-Bayes generalization …
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach b…
Image Classification