Understand the Effect of Importance Weighting in Deep Learning on Dataset Shift
We evaluate the effectiveness of importance weighting in deep neural networks under label shift and covariate shift. On synthetic 2D data (linearly separable and moon-shaped) using logistic regression and MLPs, we observe that weighting strongly affects decision boundaries early in training but fades with prolonged optimization. On CIFAR-10 with various class imbalances, only L2 regularization (not dropout) helps preserve weighting effects. In a covariate-shift experiment, importance weighting yields no significant performance gain, highlighting challenges on complex data. Our results call into question the practical utility of importance weighting for real-world distribution shifts.
Code (0)
등록된 구현이 없습니다.
Tasks
L2 RegularizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Is Importance Weighting Incompatible with Interpolating Classifiers?
Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance weights can have little to no effect on…
Nearest Neighbor-based Importance Weighting
Importance weighting is widely applicable in machine learning in general and in techniques dealing with data covariate shift problems in particular. A novel, direct approach to determine such importance weighting is pres…
BIG-bench Machine LearningClassificationGeneral ClassificationImportance Weighting Correction of Regularized Least-Squares for Covariate and Target Shifts
In many real world problems, the training data and test data have different distributions. This situation is commonly referred as a dataset shift. The most common settings for dataset shift often considered in the litera…
regressionHigh-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization
This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate …
A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization
Covariate-shift generalization, a typical case in out-of-distribution (OOD) generalization, requires a good performance on the unknown test distribution, which varies from the accessible training distribution in the form…
feature selection