Understanding Benign Overfitting in Gradient-Based Meta Learning
Meta learning has demonstrated tremendous success in few-shot learning with limited supervised data. In those settings, the meta model is usually overparameterized. While the conventional statistical learning theory suggests that overparameterized models tend to overfit, empirical evidence reveals that overparameterized meta learning methods still work well -- a phenomenon often called "benign overfitting." To understand this phenomenon, we focus on the meta learning settings with a challenging bilevel structure that we term the gradient-based meta learning, and analyze its generalization performance under an overparameterized meta linear regression model. While our analysis uses the relatively tractable linear models, our theory contributes to understanding the delicate interplay among data heterogeneity, model adaptation and benign overfitting in gradient-based meta learning tasks. We corroborate our theoretical claims through numerical simulations.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningLearning TheoryMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Benign Overfitting in Two-layer Convolutional Neural Networks
Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred to as "benign overfitting". Recently, t…
Vocal Bursts Valence PredictionA Classical View on Benign Overfitting: The Role of Sample Size
Benign overfitting is a phenomenon in machine learning where a model perfectly fits (interpolates) the training data, including noisy examples, yet still generalizes well to unseen data. Understanding this phenomenon has…
regressionTheoretical Characterization of the Generalization Performance of Overfitted Meta-Learning
Meta-learning has arisen as a successful method for improving training performance by training over many similar tasks, especially with deep neural networks (DNNs). However, the theoretical understanding of when and why …
DiversityMeta-LearningregressionBenign Overfitting under Learning Rate Conditions for $α$ Sub-exponential Input
This paper investigates the phenomenon of benign overfitting in binary classification problems with heavy-tailed input distributions, extending the analysis of maximum margin classifiers to $\alpha$ sub-exponential distr…
Binary ClassificationBenign Overfitting in Single-Head Attention
The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and…