Meta-Learning with Network Pruning
Meta-learning is a powerful paradigm for few-shot learning. Although with remarkable success witnessed in many applications, the existing optimization based meta-learning models with over-parameterized neural networks have been evidenced to ovetfit on training tasks. To remedy this deficiency, we propose a network pruning based meta-learning approach for overfitting reduction via explicitly controlling the capacity of network. A uniform concentration analysis reveals the benefit of network capacity constraint for reducing generalization gap of the proposed meta-learner. We have implemented our approach on top of Reptile assembled with two network pruning routines: Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method not only effectively alleviates meta-overfitting but also in many cases improves the overall generalization performance when applied to few-shot classification tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningMeta-LearningNetwork PruningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for an…
AutoMLMeta-LearningReal-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning
We propose a novel meta-learning framework for real-time object tracking with efficient model adaptation and channel pruning. Given an object tracker, our framework learns to fine-tune its model parameters in only a few …
Meta-LearningObjectObject TrackingVote for Nearest Neighbors Meta-Pruning of Self-Supervised Networks
Pruning plays an essential role in deploying deep neural nets (DNNs) to the hardware of limited memory or computation. However, current high-quality iterative pruning can create a terrible carbon footprint when compressi…
Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks
As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task? How to r…
Improving Generalization in Meta-Learning via Meta-Gradient Augmentation
Meta-learning methods typically follow a two-loop framework, where each loop potentially suffers from notorious overfitting, hindering rapid adaptation and generalization to new tasks. Existing schemes solve it by enhanc…
Few-Shot LearningMemorizationMeta-LearningNetwork Pruning