Provable Generalization of Overparameterized Meta-learning Trained with SGD

2022-06-18 · Yu Huang, Yingbin Liang, Longbo Huang

Despite the superior empirical success of deep meta-learning, theoretical understanding of overparameterized meta-learning is still limited. This paper studies the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML), which aims to find a good initialization for fast adaptation to new tasks. Under a mixed linear regression model, we analyze the generalization properties of MAML trained with SGD in the overparameterized regime. We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. With such sharp characterizations, we further explore how various learning parameters impact the generalization capability of overparameterized MAML, including explicitly identifying typical data and task distributions that can achieve diminishing generalization error with overparameterization, and characterizing the impact of adaptation learning rate on both excess risk and the early stopping time. Our theoretical findings are further validated by experiments.

📄 PDF Abstract BibTeX arXiv:2206.09136

Code (0)

등록된 구현이 없습니다.

Tasks

Generalization BoundsMeta-Learning

Methods 이 논문이 사용한 방법론

Early Stopping Early Stopping is a regularization technique for deep neural networks that stops training when parameter updates no longer begin to yield improves on a validation set. In…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

MAML 설명 없음

Linear Regression Linear Regression is a method for modelling a relationship between a dependent variable and independent variables. These models can be fit with numerous approaches. The most…

Understanding Benign Overfitting in Gradient-Based Meta Learning

2022-06-27 · Lisha Chen, Songtao Lu, Tianyi Chen

Meta learning has demonstrated tremendous success in few-shot learning with limited supervised data. In those settings, the meta model is usually overparameterized. While the conventional statistical learning theory sugg…

Few-Shot Learning Learning Theory Meta-Learning

Provable Generalization in Overparameterized Neural Nets

2025-08-24 · Aviral Dhingra arxiv

Deep neural networks often contain far more parameters than training examples, yet they still manage to generalize well in practice. Classical complexity measures such as VC-dimension or PAC-Bayes bounds usually become v…

Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise

2021-01-04 · Spencer Frei, Yuan Cao, Quanquan Gu

We consider a one-hidden-layer leaky ReLU network of arbitrary width trained by stochastic gradient descent (SGD) following an arbitrary initialization. We prove that SGD produces neural networks that have classification…

Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference

2022-02-02 · Jasper Tan, Blake Mason, Hamid Javadi, Richard G. Baraniuk

A surprising phenomenon in modern machine learning is the ability of a highly overparameterized model to generalize well (small error on the test data) even when it is trained to memorize the training data (zero error on…

Inference Attack Membership Inference Attack regression

Online Constrained Meta-Learning: Provable Guarantees for Generalization

2023-09-21 · NeurIPS 2023 11

Meta-learning has attracted attention due to its strong ability to learn experiences from known tasks, which can speed up and enhance the learning process for new tasks. However, most existing meta-learning approaches on…