Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models
Pre-trained masked language models successfully perform few-shot learning by formulating downstream tasks as text infilling. However, as a strong alternative in full-shot settings, discriminative pre-trained models like ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks. ELECTRA is pre-trained to distinguish if a token is generated or original. We naturally extend that to prompt-based few-shot learning by training to score the originality of the target options without introducing new parameters. Our method can be easily adapted to tasks involving multi-token predictions without extra computation overhead. Analysis shows that ELECTRA learns distributions that align better with downstream tasks.
Code (1)
Tasks
Few-Shot LearningText InfillingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ELECTRA is a Zero-Shot Learner, Too
Recently, for few-shot or even zero-shot learning, the new paradigm "pre-train, prompt, and predict" has achieved remarkable achievements compared with the "pre-train, fine-tune" paradigm. After the success of prompt-bas…
Language ModelingLanguage ModellingPrompt LearningSST-2+1Discriminative Language Model as Semantic Consistency Scorer for Prompt-based Few-Shot Text Classification
This paper proposes a novel prompt-based finetuning method (called DLM-SCS) for few-shot text classification by utilizing the discriminative language model ELECTRA that is pretrained to distinguish whether a token is ori…
Few-Shot Text ClassificationLanguage ModelingLanguage Modellingtext-classification+1On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining
Neural language representation models such as BERT have recently shown state of the art performance in downstream NLP tasks and bio-medical domain adaptation of BERT (Bio-BERT) has shown same behavior on biomedical text …
Domain AdaptationGPUnamed-entity-recognitionNamed Entity Recognition+3Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting
Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off…
Action RecognitionPrompt LearningVideo ClassificationZero-Shot Action Recognition+1Prompt Tuning for Discriminative Pre-trained Language Models
Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on pro…
Language ModelingLanguage ModellingQuestion Answeringtext-classification+1