DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.
Code (1)
Tasks
Story GenerationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
In this work, we propose to use linguistic annotations as a basis for a \textit{Discourse-Aware Semantic Self-Attention} encoder that we employ for reading comprehension on long narrative texts. We extract relations betw…
Reading ComprehensionSentenceDiscourse-Aware Neural Rewards for Coherent Text Generation
In this paper, we investigate the use of discourse-aware rewards with reinforcement learning to guide a model to generate long, coherent text. In particular, we propose to learn neural rewards to model cross-sentence ord…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Sentence+2Paragraph-level Commonsense Transformers with Recurrent Memory
Human understanding of narrative texts requires making commonsense inferences beyond what is stated explicitly in the text. A recent model, COMET, can generate such implicit commonsense inferences along several dimension…
SentenceWorld KnowledgeDiscourse-Aware Neural Extractive Text Summarization
Recently BERT has been adopted for document encoding in state-of-the-art text summarization models. However, sentence-based extractive models often result in redundant or uninformative phrases in the extracted summaries.…
Extractive Text SummarizationSentenceText SummarizationAlign to Structure: Aligning Large Language Models with Structural Information
Generating long, coherent text remains a challenge for large language models (LLMs), as they lack hierarchical planning and structured organization in discourse generation. We introduce Structural Alignment, a novel meth…
Document SummarizationText Generation