Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures
In this paper, we propose an extension to Longformer Encoder-Decoder, a popular sparse transformer architecture. One common challenge with sparse transformers is that they can struggle with encoding of long range context, such as connections between topics discussed at a beginning and end of a document. A method to selectively increase global attention is proposed and demonstrated for abstractive summarization tasks on several benchmark data sets. By prefixing the transcript with additional keywords and encoding global attention on these keywords, improvement in zero-shot, few-shot, and fine-tuned cases is demonstrated for some benchmark data sets.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationDecoderMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evolution of Event Designation in Media: Preliminary Study
Within the general purpose of information extraction, detection of event descriptions is often an important clue. An important characteristic of event designation in texts, and especially in media, is that it changes ove…
DescriptiveInformation RetrievalRetrievalEvent Nominals: Annotation Guidelines and a Manually Annotated Corpus in French
Within the general purpose of information extraction, detection of event descriptions is an important clue. A word refering to an event is more powerful than a single word, because it implies a location, a time, protagon…
ArticlesUnsupervised extraction of local and global keywords from a single text
We propose an unsupervised, corpus-independent method to extract keywords from a single text. It is based on the spatial distribution of words and the response of this distribution to a random permutation of words. As co…
Multi-Task Network for Noise-Robust Keyword Spotting and Speaker Verification using CTC-based Soft VAD and Global Query Attention
Keyword spotting (KWS) and speaker verification (SV) have been studied independently although it is known that acoustic and speaker domains are complementary. In this paper, we propose a multi-task network that performs …
Action DetectionActivity DetectionKeyword SpottingSpeaker VerificationA Global-Local Attention Mechanism for Relation Classification
Relation classification, a crucial component of relation extraction, involves identifying connections between two entities. Previous studies have predominantly focused on integrating the attention mechanism into relation…
ClassificationRelationRelation ClassificationRelation Extraction