Towards Unified Prompt Tuning for Few-shot Learning
Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot learning by employing task-specific prompts. However, PLMs are unfamiliar with the prompt-style expressions during pre-training, which limits the few-shot learning performance on downstream tasks. It would be desirable if models can acquire some prompting knowledge before task adaptation. We present the Unified Prompt Tuning (UPT) framework, leading to better few-shot learning for BERT-style models by explicitly capturing prompting semantics from non-target NLP datasets. In UPT, a novel paradigm Prompt-Options-Verbalizer is proposed for joint prompt learning across different NLP tasks, forcing PLMs to capture task-invariant prompting knowledge. We further design a self-supervised task named Knowledge-enhanced Selective Masked Language Modeling to improve the PLM's generalization abilities for accurate adaptation to previously unseen tasks. After multi-task learning, the PLM can be fine-tuned for any target few-shot NLP tasks using the same prompting paradigm. Experiments over a variety of NLP tasks show that UPT consistently outperforms state-of-the-arts for prompt-based fine-tuning.
Code (0)
등록된 구현이 없습니다.
Tasks
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language ModelingMulti-Task LearningPrompt LearningSimilar Papers 제목 키워드 기반
Few-shot Unified Question Answering: Tuning Models or Prompts?
Question-answering (QA) tasks often investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. While recent research has explor…
Question AnsweringTransfer LearningPPT: Pre-trained Prompt Tuning for Few-shot Learning
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and onl…
AttributeFew-Shot LearningPPT: Pre-trained Prompt Tuning for Few-shot Learning
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and onl…
AttributeFew-Shot LearningTowards Unified Prompt Tuning for Few-shot Text Classification
Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot text classification by employing task-specific prompts. Yet, PLMs are unfamiliar with prompt-style expressions during…
ClassificationFew-Shot LearningFew-Shot Text ClassificationLanguage Modeling+6Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt
Prompt-based tuning has been proven effective for pretrained language models (PLMs). While most of the existing work focuses on the monolingual prompts, we study the multilingual prompts for multilingual PLMs, especially…
Cross-Lingual TransferZero-Shot Cross-Lingual Transfer