ZeroGen: Efficient Zero-shot Learning via Dataset Generation
There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, \textsc{ZeroGen}. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being annotation-free and efficient, we argue that \textsc{ZeroGen} can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of \textsc{ZeroGen}.
Code (3)
Tasks
Data-free Knowledge DistillationDataset GenerationKnowledge DistillationNatural Language InferenceQuestion Answeringtext-classificationText ClassificationText GenerationZero-Shot LearningSimilar Papers 제목 키워드 기반
ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles
Automatically generating textual content with desired attributes is an ambitious task that people have pursued long. Existing works have made a series of progress in incorporating unimodal controls into language models (…
News GenerationSentenceText GenerationHyper-Parameter Sweep on AlphaZero General
Since AlphaGo and AlphaGo Zero have achieved breakground successes in the game of Go, the programs have been generalized to solve other tasks. Subsequently, AlphaZero was developed to play Go, Chess and Shogi. In the lit…
Game of GoMulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generation
Building dialogue generation systems in a zero-shot scenario remains a huge challenge, since the typical zero-shot approaches in dialogue generation rely heavily on large-scale pre-trained language generation models such…
Data AugmentationDialogue GenerationText GenerationLet's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models
*Data Synthesis* is a promising way to train a small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training example…
Language ModelingLanguage ModellingLarge Language ModelProGen: Progressive Zero-shot Dataset Generation via In-context Feedback
Recently, dataset-generation-based zero-shot learning has shown promising results by training a task-specific model with a dataset synthesized from large pre-trained language models (PLMs). The final task-specific model …
Data-free Knowledge DistillationDataset GenerationInformativenesstext-classification+2