Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners without any prompt engineering. The main principle behind this approach involves reformulating potential natural language processing tasks into the task of a pre-trained language model and differentially optimizing the prompt template as well as the target label with backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any pre-trained language models; (ii) Extended to widespread classification tasks. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves a better few-shot performance. Code is available in https://github.com/zjunlp/DART.
Code (4)
Tasks
Language ModelingLanguage ModellingPrompt EngineeringSimilar Papers 제목 키워드 기반
Differentiable Prompt Learning for Vision Language Models
Prompt learning is an effective way to exploit the potential of large-scale pre-trained foundational models. Continuous prompts parameterize context tokens in prompts by turning them into differentiable vectors. Deep con…
Prompt LearningContrastive Learning for Prompt-Based Few-Shot Language Learners
The impressive performance of GPT-3 using natural language prompts and in-context learning has inspired work on better fine-tuning of moderately-sized models under this paradigm. Following this line of work, we present a…
Contrastive LearningIn-Context LearningLanguage ModelingLanguage Modelling+1What Makes Pre-trained Language Models Better Zero-shot Learners?
Current methods for prompt learning in zeroshot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a posteriori. This is not ideal because in a r…
Language ModellingPrompt Learningtext-classificationText Classification+1An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
The increasingly large size of modern pretrained language models not only makes them inherit more human-like biases from the training corpora, but also makes it computationally expensive to mitigate such biases. In this …
counterfactualData AugmentationRetrievalOn Conditional and Compositional Language Model Differentiable Prompting
Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. Prompts can be represented by a human-engineered word sequence or by a learned cont…
Few-Shot LearningLanguage ModelingLanguage Modellingmodel+1