Self-training Avoids Using Spurious Features Under Domain Shift
In unsupervised domain adaptation, existing theory focuses on situations where the source and target domains are close. In practice, conditional entropy minimization and pseudo-labeling work even when the domain shifts are much larger than those analyzed by existing theory. We identify and analyze one particular setting where the domain shift can be large, but these algorithms provably work: certain spurious features correlate with the label in the source domain but are independent of the label in the target. Our analysis considers linear classification where the spurious features are Gaussian and the non-spurious features are a mixture of log-concave distributions. For this setting, we prove that entropy minimization on unlabeled target data will avoid using the spurious feature if initialized with a decently accurate source classifier, even though the objective is non-convex and contains multiple bad local minima using the spurious features. We verify our theory for spurious domain shift tasks on semi-synthetic Celeb-A and MNIST datasets. Our results suggest that practitioners collect and self-train on large, diverse datasets to reduce biases in classifiers even if labeling is impractical.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationUnsupervised Domain AdaptationSimilar Papers 제목 키워드 기반
Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately
The presence of spurious features interferes with the goal of obtaining robust models that perform well across many groups within the population. A natural remedy is to remove spurious features from the model. However, i…
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
Despite alarm over the reliance of machine learning systems on so-called spurious patterns, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious …
counterfactualData AugmentationNatural Language InferenceSentiment AnalysisCausally Robust Reward Learning from Reason-Augmented Preference Feedback
Preference-based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches…
Entropy Penalty: Towards Generalization Beyond the IID Assumption
It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve…
Mitigating Spurious Correlations for Self-supervised Recommendation
Recent years have witnessed the great success of self-supervised learning (SSL) in recommendation systems. However, SSL recommender models are likely to suffer from spurious correlations, leading to poor generalization. …
Feature EngineeringRecommendation SystemsSelf-Supervised Learning