DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection
Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset that reduces the amount of data needed to fine-tune PLMs for downstream tasks. We examine the efficacy of DEFT-UCS in the context of text-editing LMs, and compare to the state-of-the art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Vision-Language Models are Strong Noisy Label Detectors
Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a si…
Denoisingimage-classificationImage Classificationparameter-efficient fine-tuningFrom PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers
Pretrained Language Models (PLMs) have become the de facto starting point for fine-tuning on downstream tasks. However, as model sizes continue to increase, traditional fine-tuning of all the parameters becomes challengi…
Decoderparameter-efficient fine-tuningDeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer
Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches …
Cross-Lingual TransferNatural Language InferenceSentiment AnalysisSentiment Classification+1DEFT-VTON: Efficient Virtual Try-On with Consistent Generalised H-Transform
Diffusion models enable high-quality virtual try-on (VTO) with their established image synthesis abilities. Despite the extensive end-to-end training of large pre-trained models involved in current VTO methods, real-worl…
parameter-efficient fine-tuningVirtual Try-onUPB at SemEval-2020 Task 6: Pretrained Language Models for Definition Extraction
This work presents our contribution in the context of the 6th task of SemEval-2020: Extracting Definitions from Free Text in Textbooks (DeftEval). This competition consists of three subtasks with different levels of gran…
Definition ExtractionGeneral ClassificationLanguage ModelingLanguage Modelling+1