TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Text summarization aims to extract essential information from a piece of text and transform the text into a concise version. Existing unsupervised abstractive summarization models leverage recurrent neural networks framework while the recently proposed transformer exhibits much more capability. Moreover, most of previous summarization models ignore abundant unlabeled corpora resources available for pretraining. In order to address these issues, we propose TED, a transformer-based unsupervised abstractive summarization system with pretraining on large-scale data. We first leverage the lead bias in news articles to pretrain the model on millions of unlabeled corpora. Next, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of generated summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are highly abstractive, and each component in the objective function of TED is highly effective.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationArticlesDenoisingText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Simple Unsupervised Summarization by Contextual Matching
We propose an unsupervised method for sentence summarization using only language modeling. The approach employs two language models, one that is generic (i.e. pretrained), and the other that is specific to the target dom…
Language ModelingLanguage ModellingSentenceSentence Summarization+2NUTS: Network for Unsupervised Telegraphic Summarization
Extractive summarization methods operate by ranking and selecting the sentences which best encapsulate the theme of a given document. They do not fare well in domains like fictional narratives where there is no central t…
DecoderExtractive SummarizationSentenceVideo Summarization using Denoising Diffusion Probabilistic Model
Video summarization aims to eliminate visual redundancy while retaining key parts of video to construct concise and comprehensive synopses. Most existing methods use discriminative models to predict the importance scores…
DenoisingmodelUnsupervised Video SummarizationVideo SummarizationInvestigating the Impact of Text Summarization on Topic Modeling
Topic models are used to identify and group similar themes in a set of documents. Recent advancements in deep learning based neural topic models has received significant research interest. In this paper, an approach is p…
DiversityLanguage ModelingLanguage ModellingLarge Language Model+2Improved Spoken Document Summarization with Coverage Modeling Techniques
Extractive summarization aims at selecting a set of indicative sentences from a source document as a summary that can express the major theme of the document. A general consensus on extractive summarization is that both …
DiversityDocument SummarizationExtractive SummarizationSentence