Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
We derive a novel information-theoretic analysis of the generalization property of meta-learning algorithms. Concretely, our analysis proposes a generic understanding of both the conventional learning-to-learn framework and the modern model-agnostic meta-learning (MAML) algorithms. Moreover, we provide a data-dependent generalization bound for a stochastic variant of MAML, which is non-vacuous for deep few-shot learning. As compared to previous bounds that depend on the square norm of gradients, empirical validations on both simulated data and a well-known few-shot benchmark show that our bound is orders of magnitude tighter in most situations.
Code (1)
Tasks
Few-Shot LearningGeneralization BoundsMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Sharper Information-theoretic Generalization Bounds for Meta-Learning
In recent years, information-theoretic generalization bounds have emerged as a promising approach for analyzing the generalization capabilities of meta-learning algorithms. However, existing results are confined to two-s…
Generalization BoundsMeta-LearningAn Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learning
In this work, we study out-of-distribution (OOD) generalization in meta-reinforcement learning from an information-theoretic perspective. We begin by establishing OOD generalization bounds for meta-supervised learning un…
Reinforcement LearningAn Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning
Meta-learning aims at optimizing the hyperparameters of a model class or training algorithm from the observation of data from a number of related tasks. Following the setting of Baxter [1], the tasks are assumed to belon…
Meta-LearningInformation-Theoretic Generalization Bounds for Meta-Learning and Applications
Meta-learning, or "learning to learn", refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, …
Generalization BoundsInductive BiasMeta-LearningSample Compression Hypernetworks: From Generalization Bounds to Meta-Learning
Reconstruction functions are pivotal in sample compression theory, a framework for deriving tight generalization bounds. From a small sample of the training set (the compression set) and an optional stream of information…
Generalization BoundsMeta-Learning