Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift
Many machine learning models appear to deploy effortlessly under distribution shift, and perform well on a target distribution that is considerably different from the training distribution. Yet, learning theory of distribution shift bounds performance on the target distribution as a function of the discrepancy between the source and target, rarely guaranteeing high target accuracy. Motivated by this gap, this work takes a closer look at the theory of distribution shift for a classifier from a source to a target distribution. Instead of relying on the discrepancy, we adopt an Invariant-Risk-Minimization (IRM)-like assumption connecting the distributions, and characterize conditions under which data from a source distribution is sufficient for accurate classification of the target. When these conditions are not met, we show when only unlabeled data from the target is sufficient, and when labeled target data is needed. In all cases, we provide rigorous theoretical guarantees in the large sample regime.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning TheorySimilar Papers 제목 키워드 기반
A Closer Look at the Learnability of Out-of-Distribution (OOD) Detection
Machine learning algorithms often encounter different or "out-of-distribution" (OOD) data at deployment time, and OOD detection is frequently employed to detect these examples. While it works reasonably well in practice,…
Learning TheoryOut of Distribution (OOD) DetectionPAC learningBeyond AUROC & co. for evaluating out-of-distribution detection performance
While there has been a growing research interest in developing out-of-distribution (OOD) detection methods, there has been comparably little discussion around how these methods should be evaluated. Given their relevance …
Binary ClassificationOut-of-Distribution DetectionOut of Distribution (OOD) DetectionOne-Shot Decision-Making with and without Surrogates
One-shot decision making is required in situations in which we can evaluate a fixed number of solution candidates but do not have any possibility for further, adaptive sampling. Such settings are frequently encountered i…
Decision MakingregressionCycle Consistent Probability Divergences Across Different Spaces
Discrepancy measures between probability distributions are at the core of statistical inference and machine learning. In many applications, distributions of interest are supported on different spaces, and yet a meaningfu…
Computational EfficiencyGenerative Adversarial NetworkH-divergence: A Decision-Theoretic Discrepancy Measure for Two Sample Tests
Measuring the discrepancy between two probability distributions is a fundamental problem in machine learning and statistics. Based on ideas from decision theory, we investigate a new class of discrepancies that are based…
Vocal Bursts Valence Prediction