Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
Transfer learning has emerged as a powerful technique for improving the performance of machine learning models on new domains where labeled training data may be scarce. In this approach a model trained for a source task, where plenty of labeled training data is available, is used as a starting point for training a model on a related target task with only few labeled training data. Despite recent empirical success of transfer learning approaches, the benefits and fundamental limits of transfer learning are poorly understood. In this paper we develop a statistical minimax framework to characterize the fundamental limits of transfer learning in the context of regression with linear and one-hidden layer neural network models. Specifically, we derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data as well as appropriate notions of similarity between the source and target tasks. Our lower bound provides new insights into the benefits and limitations of transfer learning. We further corroborate our theoretical finding with various experiments.
Code (2)
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
The Cost of Privacy in Generalized Linear Models: Algorithms and Minimax Lower Bounds
We propose differentially private algorithms for parameter estimation in both low-dimensional and high-dimensional sparse generalized linear models (GLMs) by constructing private versions of projected gradient descent. W…
LEMMAparameter estimationHigh-probability minimax lower bounds
The minimax risk is often considered as a gold standard against which we can compare specific statistical procedures. Nevertheless, as has been observed recently in robust and heavy-tailed estimation problems, the inhere…
Density EstimationregressionMinimax optimal transfer learning for high-dimensional additive regression
This paper studies high-dimensional additive regression under the transfer learning framework, where one observes samples from a target population together with auxiliary samples from different but potentially related re…
Transfer LearningTransfer Faster, Price Smarter: Minimax Dynamic Pricing under Cross-Market Preference Shift
We study contextual dynamic pricing when a target market can leverage K auxiliary markets -- offline logs or concurrent streams -- whose mean utilities differ by a structured preference shift. We propose Cross-Market Tra…
Transfer LearningA Class of Geometric Structures in Transfer Learning: Minimax Bounds and Optimality
We study the problem of transfer learning, observing that previous efforts to understand its information-theoretic limits do not fully exploit the geometric structure of the source and target domains. In contrast, our st…
Transfer Learning