Controlling Pretrained Language Generation Models by Learning to Focus
Transformer-based language models, which are pretrained on large-scale unsupervised data and then finetuned on task-specific datasets, have become the dominant paradigm for various natural language generation tasks. The finetuning and usages of such models are typically conducted in an end-to-end manner. This work attempts to develop a control mechanism by which a user can select spans of context as "highlights'' for the model to focus on, while generating output text. To achieve this goal, we augment a pretrained model with trainable "attention vectors'' that are directly applied to the model's embeddings, while the model itself is kept fixed. These vectors, trained on automatic annotations derived from attribution methods, act as indicators for context importance. We test our approach on two core generation tasks: dialogue response generation and abstractive summarization. We also collect evaluation data where the highlight-generation pairs are annotated by humans. Our experiments show that the trained attention vectors are effective in steering the model to generate outputs that are relevant to user-selected highlights.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationResponse GenerationText GenerationSimilar Papers 제목 키워드 기반
Personalized Text Generation with Fine-Grained Linguistic Control
As the text generation capabilities of large language models become increasingly prominent, recent studies have focused on controlling particular aspects of the generated text to make it more personalized. However, most …
Text GenerationControlling the Focus of Pretrained Language Generation Models
The finetuning of pretrained transformer-based language generation models are typically conducted in an end-to-end manner, where the model learns to attend to relevant parts of the input by itself. However, there does no…
Abstractive Text SummarizationResponse GenerationText GenerationControlling the Focus of Pretrained Language Generation Models
The finetuning of pretrained transformer-based language generation models are typically conducted in an end-to-end manner, where the model learns to attend to relevant parts of the input by itself. However, there does no…
Abstractive Text SummarizationResponse GenerationText GenerationTowards Neural Programming Interfaces
It is notoriously difficult to control the behavior of artificial neural networks such as generative neural language models. We recast the problem of controlling natural language generation as that of learning to interfa…
Language ModelingLanguage ModellingText GenerationUsing Structured Content Plans for Fine-grained Syntactic Control in Pretrained Language Model Generation
Large pretrained language models offer powerful generation capabilities, but suffer from a lack of interpretability and fine-grained control. We propose an approach to fine-grained control in generating text directly fr…
Abstract Meaning RepresentationLanguage ModelingLanguage ModellingSentence+1