Self-Supervised Learning for Contextualized Extractive Summarization
Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level. In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion. Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks. Furthermore, we show that after pre-training, a clean model with simple building blocks is able to outperform previous state-of-the-art that are carefully designed.
Code (2)
Tasks
Extractive SummarizationSelf-Supervised LearningSimilar Papers 제목 키워드 기반
A General Contextualized Rewriting Framework for Text Summarization
The rewriting method for text summarization combines extractive and abstractive approaches, improving the conciseness and readability of extractive summaries using an abstractive model. Exiting rewriting systems take eac…
reinforcement-learningReinforcement Learning (RL)SentenceTAG+1Contextualized Rewriting for Text Summarization
Extractive summarization suffers from irrelevance, redundancy and incoherence. Existing work shows that abstractive rewriting for extractive summaries can improve the conciseness and readability. These rewriting systems …
Extractive Summarizationreinforcement-learningReinforcement Learning (RL)TAG+1The Effect of Pretraining on Extractive Summarization for Scientific Documents
Large pretrained models have seen enormous success in extractive summarization tasks. In this work, we investigate the influence of pretraining on a BERT-based extractive summarization system for scientific documents. We…
Extractive SummarizationWord EmbeddingsBottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle
The principle of the Information Bottleneck (Tishby et al. 1999) is to produce a summary of information X optimized to predict some other relevant information Y. In this paper, we propose a novel approach to unsupervised…
Abstractive Text SummarizationExtractive SummarizationLanguage ModelingLanguage Modelling+4Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge …
Document SummarizationExtractive Document SummarizationExtractive SummarizationExtractive Text Summarization+2