Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias
We characterize and study zero-shot abstractive summarization in Large Language Models (LLMs) by measuring position bias, which we propose as a general formulation of the more restrictive lead bias phenomenon studied previously in the literature. Position bias captures the tendency of a model unfairly prioritizing information from certain parts of the input text over others, leading to undesirable behavior. Through numerous experiments on four diverse real-world datasets, we study position bias in multiple LLM models such as GPT 3.5-Turbo, Llama-2, and Dolly-v2, as well as state-of-the-art pretrained encoder-decoder abstractive summarization models such as Pegasus and BART. Our findings lead to novel insights and discussion on performance and position bias of models for zero-shot summarization tasks.
Code (1)
Tasks
Abstractive Text SummarizationDecoderPositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Restructuring Conversations using Discourse Relations for Zero-shot Abstractive Dialogue Summarization
Dialogue summarization is a challenging problem due to the informal and unstructured nature of conversational data. Recent advances in abstractive summarization have been focused on data-hungry neural models and adapting…
Abstractive Dialogue SummarizationAbstractive Text SummarizationDocument SummarizationFlight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization
Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks including automatic summarization. However, most work has focused on (relatively) data-rich sing…
Abstractive Text SummarizationDocument SummarizationFew-Shot LearningMulti-Document SummarizationImproving the Faithfulness of Abstractive Summarization via Entity Coverage Control
Abstractive summarization systems leveraging pre-training language models have achieved superior results on benchmark datasets. However, such models have been shown to be more prone to hallucinate facts that are unfaithf…
Abstractive Text SummarizationImproving the Faithfulness of Abstractive Summarization via Entity Coverage Control
Abstractive summarization systems leveraging pre-training language models have achieved superior results on benchmark datasets. However, such models have been shown to be more prone to hallucinate facts that are unfaithf…
Abstractive Text SummarizationAssessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing
Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles. However, little is known about the robustness of such a process of zero-shot su…
Abstractive Text SummarizationArticles