Testing for concept shift online
This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionSimilar Papers 제목 키워드 기반
Learning to Adapt to Online Streams with Distribution Shifts
Test-time adaptation (TTA) is a technique used to reduce distribution gaps between the training and testing sets by leveraging unlabeled test data during inference. In this work, we expand TTA to a more practical scenari…
BenchmarkingMeta-LearningTest-time AdaptationVideo Segmentation+1Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
Moving beyond testing on in-distribution data works on Out-of-Distribution (OOD) detection have recently increased in popularity. A recent attempt to categorize OOD data introduces the concept of near and far OOD detecti…
Out of Distribution (OOD) DetectionSensitivityExploring Covariate and Concept Shift for Detection and Confidence Calibration of Out-of-Distribution Data
Moving beyond testing on in-distribution data, works on Out-of-Distribution (OOD) detection have recently increased in popularity. A recent attempt to categorize OOD data introduces the concept of near and far OOD detect…
Out of Distribution (OOD) DetectionSensitivityAdapting to Online Label Shift with Provable Guarantees
The standard supervised learning paradigm works effectively when training data shares the same distribution as the upcoming testing samples. However, this stationary assumption is often violated in real-world application…
Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically requir…