Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study
Text summarization is a downstream natural language processing (NLP) task that challenges the understanding and generation capabilities of language models. Considerable progress has been made in automatically summarizing short texts, such as news articles, often leading to satisfactory results. However, summarizing long documents remains a major challenge. This is due to the complex contextual information in the text and the lack of open-source benchmarking datasets and evaluation frameworks that can be used to develop and test model performance. In this work, we use ChatGPT, the latest breakthrough in the field of large language models (LLMs), together with the extractive summarization model C2F-FAR (Coarse-to-Fine Facet-Aware Ranking) to propose a hybrid extraction and summarization pipeline for long documents such as business articles and books. We work with the world-renowned company getAbstract AG and leverage their expertise and experience in professional book summarization. A practical study has shown that machine-generated summaries can perform at least as well as human-written summaries when evaluated using current automated evaluation metrics. However, a closer examination of the texts generated by ChatGPT through human evaluations has shown that there are still critical issues in terms of text coherence, faithfulness, and style. Overall, our results show that the use of ChatGPT is a very promising but not yet mature approach for summarizing long documents and can at best serve as an inspiration for human editors. We anticipate that our work will inform NLP researchers about the extent to which ChatGPT's capabilities for summarizing long documents overlap with practitioners' needs. Further work is needed to test the proposed hybrid summarization pipeline, in particular involving GPT-4, and to propose a new evaluation framework tailored to the task of summarizing long documents.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesBenchmarkingBook summarizationDocument SummarizationExtractive SummarizationText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Novel LLM-based Two-stage Summarization Approach for Long Dialogues
Long document summarization poses a significant challenge in natural language processing due to input lengths that exceed the capacity of most state-of-the-art pre-trained language models. This study proposes a hierarchi…
Abstractive Text SummarizationDocument SummarizationExploring the Limits of ChatGPT for Query or Aspect-based Text Summarization
Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various m…
Abstractive Text SummarizationArticlesText SummarizationExtractive Summarization via ChatGPT for Faithful Summary Generation
Extractive summarization is a crucial task in natural language processing that aims to condense long documents into shorter versions by directly extracting sentences. The recent introduction of large language models has …
Extractive SummarizationIn-Context LearningText SummarizationGenCompareSum: a hybrid unsupervised summarization method using salience
Text summarization (TS) is an important NLP task. Pre-trained Language Models (PLMs) have been used to improve the performance of TS. However, PLMs are limited by their need of labelled training data and by their attenti…
Extractive SummarizationText SummarizationEfficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
Extractive summarization of long documents is bottlenecked by quadratic complexity, often forcing truncation and limiting deployment in resource-constrained settings. We introduce the first Mamba-Transformer hybrid for e…