Retrain or not retrain: Conformal test martingales for change-point detection
We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Change Point DetectionConformal PredictionPredictionSimilar Papers 제목 키워드 기반
Betting on Moments: Legendre Jumper Martingales for Online Exchangeability Testing
A fundamental assumption in statistics and machine learning is that ``the future looks like the past,'' formalized as exchangeability: the joint data distribution is order-invariant. In practice, this assumption is often…
Testing randomness
The hypothesis of randomness is fundamental in statistical machine learning and in many areas of nonparametric statistics; it says that the observations are assumed to be independent and coming from the same unknown prob…
Conformal testing: binary case with Markov alternatives
We continue study of conformal testing in binary model situations. In this note we consider Markov alternatives to the null hypothesis of exchangeability. We propose two new classes of conformal test martingales; one cla…
Computational EfficiencyConformal testing in a binary model situation
Conformal testing is a way of testing the IID assumption based on conformal prediction. The topic of this note is computational evaluation of the performance of conformal testing in a model situation in which IID binary …
Conformal PredictionmodelInductive Conformal Martingales for Change-Point Detection
We consider the problem of quickest change-point detection in data streams. Classical change-point detection procedures, such as CUSUM, Shiryaev-Roberts and Posterior Probability statistics, are optimal only if the chang…
Change Point Detection