Projected Subnetworks Scale Adaptation
Large models support great zero-shot and few-shot capabilities. However, updating these models on new tasks can break performance on previous seen tasks and their zero/few-shot unseen tasks. Our work explores how to update zero/few-shot learners such that they can maintain performance on seen/unseen tasks of previous tasks as well as new tasks. By manipulating the parameter updates of a gradient-based meta learner as the projected task-specific subnetworks, we show improvements for large models to retain seen and zero/few shot task performance in online settings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Quest for Winning Tickets in Low-Rank Adapters
The Lottery Ticket Hypothesis (LTH) suggests that over-parameterized neural networks contain sparse subnetworks ("winning tickets") capable of matching full model performance when trained from scratch. With the growing r…
parameter-efficient fine-tuningTransfer LearningDeep Defocus Map Estimation Using Domain Adaptation
In this paper, we propose the first end-to-end convolutional neural network (CNN) architecture, Defocus Map Estimation Network (DMENet), for spatially varying defocus map estimation. To train the network, we produce a no…
DeblurringDefocus EstimationDomain AdaptationLess Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension
In this paper, we propose a simple few-shot domain adaptation paradigm for reading comprehension. We first identify the lottery subnetwork structure within the Transformer-based source domain model via gradual magnitude …
Domain AdaptationReading ComprehensionBias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Models
The issue of algorithmic biases in deep learning has led to the development of various debiasing techniques, many of which perform complex training procedures or dataset manipulation. However, an intriguing question aris…
Computational EfficiencyMichaelis-Menten dynamics in protein subnetworks
To understand the behaviour of complex systems it is often necessary to use models that describe the dynamics of subnetworks. It has previously been established using projection methods that such subnetwork dynamics gene…