HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Summarization systems make numerous "decisions" about summary properties during inference, e.g. degree of copying, specificity and length of outputs, etc. However, these are implicitly encoded within model parameters and specific styles cannot be enforced. To address this, we introduce HydraSum, a new summarization architecture that extends the single decoder framework of current models to a mixture-of-experts version with multiple decoders. We show that HydraSum's multiple decoders automatically learn contrasting summary styles when trained under the standard training objective without any extra supervision. Through experiments on three summarization datasets (CNN, Newsroom and XSum), we show that HydraSum provides a simple mechanism to obtain stylistically-diverse summaries by sampling from either individual decoders or their mixtures, outperforming baseline models. Finally, we demonstrate that a small modification to the gating strategy during training can enforce an even stricter style partitioning, e.g. high- vs low-abstractiveness or high- vs low-specificity, allowing users to sample from a larger area in the generation space and vary summary styles along multiple dimensions.
Code (1)
Tasks
Abstractive Text SummarizationDecoderMixture-of-ExpertsSpecificityText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
HydraSum - Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs. This results in generating generic summaries that do not cater t…
Abstractive Text SummarizationDecoderMixture-of-ExpertsSpecificity+1Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?
In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries su…
Language ModelingLanguage ModellingLarge Language ModelReview GenerationBOOKSUM: A Collection of Datasets for Long-form Narrative Summarization
The majority of existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such …
Abstractive Text SummarizationFormLong-Form Narrative SummarizationText SummarizationFourier-Attentive Representation Learning: A Fourier-Guided Framework for Few-Shot Generalization in Vision-Language Models
Large-scale pre-trained Vision-Language Models (VLMs) have demonstrated strong few-shot learning capabilities. However, these methods typically learn holistic representations where an image's domain-invariant structure i…
Representation LearningFew-Shot LearningBookSum: A Collection of Datasets for Long-form Narrative Summarization
The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such …
Abstractive Text SummarizationFormLong-Form Narrative SummarizationText Summarization