Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings
Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.
Code (1)
Tasks
Active LearningLanguage ModelingLanguage Modellingparameter-efficient fine-tuningtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning
Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has e…
Active LearningLanguage ModelingLanguage ModellingNatural Language UnderstandingUnified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning
Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up th…
In-Context LearningLanguage ModelingLanguage ModellingLarge Language Model+6Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR
Automatic speech recognition (ASR) for low-resource languages remains a challenge due to the scarcity of labeled training data. Parameter-efficient fine-tuning and text-only adaptation are two popular methods that have b…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Cross-Lingual Transferparameter-efficient fine-tuning+2Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with mil…
parameter-efficient fine-tuningJudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew
Despite significant advances in large language models, personalizing them for individual decision-makers remains an open problem. Here, we introduce a synthetic-organic supervision pipeline that transforms raw judicial d…
parameter-efficient fine-tuningSemantic Similarity