Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.
Code (2)
Tasks
Few-Shot LearningPrompt EngineeringSimilar Papers 제목 키워드 기반
STT: Soft Template Tuning for Few-Shot Learning
With the rapid expansion of large pre-trained language models, fine-tuning all the model parameters for downstream tasks is becoming computationally prohibitive. The recently developed prompt-based methods freeze the ent…
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Revisiting Automated Prompting: Are We Actually Doing Better?
Current literature demonstrates that Large Language Models (LLMs) are great few-shot learners, and prompting significantly increases their performance on a range of downstream tasks in a few-shot learning setting. An att…
Few-Shot LearningMultitask Pre-training of Modular Prompt for Chinese Few-Shot Learning
Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. Although prompt tuning has been shown to match the performance of full model tuning when training data is suffi…
Few-Shot LearningMachine Reading ComprehensionReading ComprehensionSTT: Soft Template Tuning for Few-Shot Adaptation
Prompt tuning has been an extremely effective tool to adapt a pre-trained model to downstream tasks. However, standard prompt-based methods mainly consider the case of sufficient data of downstream tasks. It is still unc…
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2The Power of Scale for Parameter-Efficient Prompt Tuning
In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GP…
Few-Shot Learning