Which Pretrain Samples to Rehearse when Finetuning Pretrained Models?
Fine-tuning pretrained foundational models on specific tasks is now the de facto approach for text and vision tasks. A known pitfall of this approach is the forgetting of pretraining knowledge that happens during finetuning. Rehearsing samples randomly from the pretrain dataset is a common approach to alleviate such forgetting. However, we find that random mixing unintentionally includes samples which are not (yet) forgotten or unlearnable by the model. We propose a novel sampling scheme, mix-cd, that identifies and prioritizes samples that actually face forgetting, which we call collateral damage. Since directly identifying collateral damage samples is computationally expensive, we propose a procedure to estimate the distribution of such samples by tracking the statistics of finetuned samples. Our approach is lightweight, easy to implement, and can be seamlessly integrated into existing models, offering an effective means to retain pretrain performance without additional computational costs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Are Fewer Labels Possible for Few-shot Learning?
Few-shot learning is challenging due to its very limited data and labels. Recent studies in big transfer (BiT) show that few-shot learning can greatly benefit from pretraining on large scale labeled dataset in a differen…
ClusteringFew-Shot LearningWhen, Why, and Which Pretrained GANs Are Useful?
The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-data regime. However, de…
Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm
Given the large-scale data and the high annotation cost, pretraining-finetuning becomes a popular paradigm in multiple computer vision tasks. Previous research has covered both the unsupervised pretraining and supervised…
Diversityimage-classificationImage ClassificationSemantic SegmentationInvestigating Transferability in Pretrained Language Models
How does language model pretraining help transfer learning? We consider a simple ablation technique for determining the impact of each pretrained layer on transfer task performance. This method, partial reinitialization,…
Language ModelingLanguage ModellingTransfer LearningActiveDC: Distribution Calibration for Active Finetuning
The pretraining-finetuning paradigm has gained popularity in various computer vision tasks. In this paradigm, the emergence of active finetuning arises due to the abundance of large-scale data and costly annotation requi…
image-classificationImage Classification