Summarization is (Almost) Dead
How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Text SummarizationSimilar Papers 제목 키워드 기반
Understanding Code Semantics: An Evaluation of Transformer Models in Summarization
This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable n…
Code SummarizationRevisiting File Context for Source Code Summarization
Source code summarization is the task of writing natural language descriptions of source code. A typical use case is generating short summaries of subroutines for use in API documentation. The heart of almost all current…
Code SummarizationDecoderSource Code SummarizationSummarization is Not Dead Yet
The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem…
Normalisation is dead, long live normalisation!
Since the advent of Batch Normalisation (BN) almost every state-of-the-art (SOTA) method uses some form of normalisation. After all, normalisation generally speeds up learning and leads to models that generalise better t…
FormIndoSum: A New Benchmark Dataset for Indonesian Text Summarization
Automatic text summarization is generally considered as a challenging task in the NLP community. One of the challenges is the publicly available and large dataset that is relatively rare and difficult to construct. The p…
ArticlesExtractive SummarizationText Summarization