Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization
A common explanation for the failure of deep networks to generalize out-of-distribution is that they fail to recover the "correct" features. We challenge this notion with a simple experiment which suggests that ERM already learns sufficient features and that the current bottleneck is not feature learning, but robust regression. Our findings also imply that given a small amount of data from the target distribution, retraining only the last linear layer will give excellent performance. We therefore argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. Towards this end, we introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Rather than learning one function, DARE performs a domain-specific adjustment to unify the domains in a canonical latent space and learns to predict in this space. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions. Further, we provide the first finite-environment convergence guarantee to the minimax risk, improving over existing analyses which only yield minimax predictors after an environment threshold. Evaluated on finetuned features, we find that DARE compares favorably to prior methods, consistently achieving equal or better performance.
Code (2)
Tasks
Domain GeneralizationOut-of-Distribution GeneralizationregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adjusted Wasserstein Distributionally Robust Estimator in Statistical Learning
We propose an adjusted Wasserstein distributionally robust estimator -- based on a nonlinear transformation of the Wasserstein distributionally robust (WDRO) estimator in statistical learning. The classic WDRO estimator …
regressionStatistical Inference for Sequential Feature Selection after Domain Adaptation
In high-dimensional regression, feature selection methods, such as sequential feature selection (SeqFS), are commonly used to identify relevant features. When data is limited, domain adaptation (DA) becomes crucial for t…
Domain Adaptationfeature selectionModel SelectionSubstitute adjustment via recovery of latent variables
The deconfounder was proposed as a method for estimating causal parameters in a context with multiple causes and unobserved confounding. It is based on recovery of a latent variable from the observed causes. We disentang…
regressionRegression-Adjusted Estimation of Quantile Treatment Effects under Covariate-Adaptive Randomizations
Datasets from field experiments with covariate-adaptive randomizations (CARs) usually contain extra covariates in addition to the strata indicators. We propose to incorporate these additional covariates via auxiliary reg…
regressionExplaining human body responses in random vibration: Effect of motion direction, sitting posture, and anthropometry
This study investigates the effects of anthropometric attributes, biological sex, and posture on translational body kinematic responses in translational vibrations. In total, 35 participants were recruited. Perturbations…
regression