PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation
Despite recent progress of pre-trained language models on generating fluent text, existing methods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow. In this work, we propose PLANET, a novel generation framework leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. To guide the generation of output sentences, our framework enriches the Transformer decoder with latent representations to maintain sentence-level semantic plans grounded by bag-of-words. Moreover, we introduce a new coherence-based contrastive learning objective to further improve the coherence of output. Extensive experiments are conducted on two challenging long-form text generation tasks including counterargument generation and opinion article generation. Both automatic and human evaluations show that our method significantly outperforms strong baselines and generates more coherent texts with richer contents.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDecoderFormSentenceText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning
Inspired by the success of the Transformer architecture in natural language processing and computer vision, we investigate the use of Transformers in Reinforcement Learning (RL), specifically in modeling the environment'…
continuous-controlContinuous ControlModel Predictive ControlQ-Learning+1PlaNet-ClothPick: Effective Fabric Flattening Based on Latent Dynamic Planning
Why do Recurrent State Space Models such as PlaNet fail at cloth manipulation tasks? Recent work has attributed this to the blurry prediction of the observation, which makes it difficult to plan directly in the latent sp…
State Space ModelsDiverse Image Inpainting with Bidirectional and Autoregressive Transformers
Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) c…
DiversityImage InpaintingLanguage ModelingLanguage ModellingLearning Latent Dynamics for Planning from Pixels
Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning…
continuous-controlContinuous ControlMotion PlanningVariational InferenceVideo-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation
Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic…
Music Generation