Hierarchical Learning for Generation with Long Source Sequences
One of the challenges for current sequence to sequence (seq2seq) models is processing long sequences, such as those in summarization and document level machine translation tasks. These tasks require the model to reason at the token level as well as the sentence and paragraph level. We design and study a new Hierarchical Attention Transformer-based architecture (HAT) that outperforms standard Transformers on several sequence to sequence tasks. Furthermore, our model achieves state-of-the-art ROUGE scores on four summarization tasks, including PubMed, arXiv, CNN/DM, SAMSum, and AMI. Our model outperforms document-level machine translation baseline on the WMT20 English to German translation task. We investigate what the hierarchical layers learn by visualizing the hierarchical encoder-decoder attention. Finally, we study hierarchical learning on encoder-only pre-training and analyze its performance on classification tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderDocument Level Machine TranslationDocument SummarizationDocument TranslationGeneral ClassificationMachine TranslationReading ComprehensionSentenceText SummarizationTranslationSimilar Papers 제목 키워드 기반
Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models
This paper presents a novel approach to generating the 3D motion of a human interacting with a target object, with a focus on solving the challenge of synthesizing long-range and diverse motions, which could not be fulfi…
DiversityHuman-Object Interaction DetectionMotion GenerationCoarse-to-Fine Attention Models for Document Summarization
Sequence-to-sequence models with attention have been successful for a variety of NLP problems, but their speed does not scale well for tasks with long source sequences such as document summarization. We propose a novel c…
Document SummarizationMachine TranslationQuestion AnsweringSampleRNN: An Unconditional End-to-End Neural Audio Generation Model
In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive mu…
Audio GenerationSpeech SynthesisTemporal SequencesTrainable Log-linear Sparse Attention for Efficient Diffusion Transformers
Diffusion Transformers (DiTs) set the state of the art in visual generation, yet their quadratic self-attention cost fundamentally limits scaling to long token sequences. Recent Top-K sparse attention approaches reduce t…
Image GenerationHierarchical Transformers Are More Efficient Language Models
Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-…
Image GenerationLanguage ModelingLanguage Modelling