TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression
The main challenge that sets transfer learning apart from traditional supervised learning is the distribution shift, reflected as the shift between the source and target models and that between the marginal covariate distributions. In this work, we tackle model shifts in the presence of covariate shifts in the high-dimensional regression setting. Specifically, we propose a two-step method with a novel fused-regularizer that effectively leverages samples from source tasks to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model, showing the robustness of the proposed method to covariate shifts. We further establish conditions under which the estimator is minimax-optimal. Additionally, we extend the method to a distributed setting, allowing for a pretraining-finetuning strategy, requiring just one round of communication while retaining the estimation rate of the centralized version. Numerical tests validate our theory, highlighting the method's robustness to covariate shifts.
Code (1)
Tasks
regressionTransfer LearningSimilar Papers 제목 키워드 기반
Transfer Learning for High-dimensional Quantile Regression with Distribution Shift
Information from related source studies can often enhance the findings of a target study. However, the distribution shift between target and source studies can severely impact the efficiency of knowledge transfer. In the…
quantile regressionregressionTransfer LearningPrecise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers
The problem of learning one task with samples from another task is central to transfer learning (TL). In this paper, we examine a fundamental question: When does combining the data samples from a source task and a target…
Multi-Task Learningtext-classificationText ClassificationTransfer LearningMinimum-Norm Interpolation Under Covariate Shift
Transfer learning is a critical part of real-world machine learning deployments and has been extensively studied in experimental works with overparameterized neural networks. However, even in the simplest setting of line…
regressionTransfer LearningEffective Sample Size, Dimensionality, and Generalization in Covariate Shift Adaptation
In supervised learning, training and test datasets are often sampled from distinct distributions. Domain adaptation techniques are thus required. Covariate shift adaptation yields good generalization performance when dom…
Dimensionality ReductionDomain Adaptationfeature selectionGeneralization BoundsTransfer Learning in Latent Contextual Bandits with Covariate Shift Through Causal Transportability
Transferring knowledge from one environment to another is an essential ability of intelligent systems. Nevertheless, when two environments are different, naively transferring all knowledge may deteriorate the performance…
Causal InferenceMulti-Armed BanditsTransfer Learning