Efficient Attentions for Long Document Summarization
The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.
Code (1)
Tasks
DecoderDocument SummarizationSimilar Papers 제목 키워드 기반
Globalizing BERT-based Transformer Architectures for Long Document Summarization
Fine-tuning a large language model on downstream tasks has become a commonly adopted process in the Natural Language Processing (NLP) (CITATION). However, such a process, when associated with the current transformer-base…
ArticlesDocument SummarizationExtractive SummarizationLanguage Modeling+2HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization
To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computati…
Document SummarizationExtractive SummarizationMulti-Document SummarizationUnsupervised Extractive Summarization by Pre-training Hierarchical Transformers
Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge …
Document SummarizationExtractive Document SummarizationExtractive SummarizationExtractive Text Summarization+2AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content
Long document summarization systems are critical for domains with lengthy and jargonladen text, yet they present significant challenges to researchers and developers with limited computing resources. Existing solutions m…
Document Summarizationdocument understandingGPUInformativenessSpeech Summarization using Restricted Self-Attention
Speech summarization is typically performed by using a cascade of speech recognition and text summarization models. End-to-end modeling of speech summarization models is challenging due to memory and compute constraints …
Document Summarizationspeech-recognitionSpeech RecognitionSpoken Language Understanding+1