PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization
Few-shot abstractive summarization has become a challenging task in natural language generation. To support it, we designed a novel soft prompts architecture coupled with a prompt pre-training plus fine-tuning paradigm that is effective and tunes only extremely light parameters. The soft prompts include continuous input embeddings across an encoder and a decoder to fit the structure of the generation models. Importantly, a novel inner-prompt placed in the text is introduced to capture document-level information. The aim is to devote attention to understanding the document that better prompts the model to generate document-related content. The first step in the summarization procedure is to conduct prompt pre-training with self-supervised pseudo-data. This teaches the model basic summarizing capabilities. The model is then fine-tuned with few-shot examples. Experimental results on the CNN/DailyMail and XSum datasets show that our method, with only 0.1% of the parameters, outperforms full-model tuning where all model parameters are tuned. It also surpasses Prompt Tuning by a large margin and delivers competitive results against Prefix-Tuning with 3% of the parameters.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationDecoderText GenerationSimilar Papers 제목 키워드 기반
PromptSum: Parameter-Efficient Controllable Abstractive Summarization
Prompt tuning (PT), a parameter-efficient technique that only tunes the additional prompt embeddings while keeping the backbone pre-trained language model (PLM) frozen, has shown promising results in language understandi…
Abstractive Text SummarizationLanguage ModelingLanguage ModellingFlight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization
Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks including automatic summarization. However, most work has focused on (relatively) data-rich sing…
Abstractive Text SummarizationDocument SummarizationFew-Shot LearningMulti-Document SummarizationRevisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias
We characterize and study zero-shot abstractive summarization in Large Language Models (LLMs) by measuring position bias, which we propose as a general formulation of the more restrictive lead bias phenomenon studied pre…
Abstractive Text SummarizationDecoderPositionA Tree-of-Thoughts Inspired Hybrid Approach for Legal Case Judgement Summarization using LLMs
In recent times, Large Language Models (LLMs) are increasingly being used for legal case judgement summarization. Most prior works have tried traditional extractive and abstractive summarization of case judgements. Howev…
OpineSum: Entailment-based self-training for abstractive opinion summarization
A typical product or place often has hundreds of reviews, and summarization of these texts is an important and challenging problem. Recent progress on abstractive summarization in domains such as news has been driven by …
Abstractive Text SummarizationArticlesFew-Shot LearningNatural Language Inference+1