Benchmarking LLMs on the Semantic Overlap Summarization Task
Semantic Overlap Summarization (SOS) is a constrained multi-document summarization task, where the constraint is to capture the common/overlapping information between two alternative narratives. While recent advancements in Large Language Models (LLMs) have achieved superior performance in numerous summarization tasks, a benchmarking study of the SOS task using LLMs is yet to be performed. As LLMs' responses are sensitive to slight variations in prompt design, a major challenge in conducting such a benchmarking study is to systematically explore a variety of prompts before drawing a reliable conclusion. Fortunately, very recently, the TELeR taxonomy has been proposed which can be used to design and explore various prompts for LLMs. Using this TELeR taxonomy and 15 popular LLMs, this paper comprehensively evaluates LLMs on the SOS Task, assessing their ability to summarize overlapping information from multiple alternative narratives. For evaluation, we report well-established metrics like ROUGE, BERTscore, and SEM-F1$ on two different datasets of alternative narratives. We conclude the paper by analyzing the strengths and limitations of various LLMs in terms of their capabilities in capturing overlapping information The code and datasets used to conduct this study are available at https://anonymous.4open.science/r/llm_eval-E16D.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingDocument SummarizationMulti-Document SummarizationSimilar Papers 제목 키워드 기반
Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization because they better capture semantic information, are better at reasoning, and are more r…
Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study
In this paper, we introduce an important yet relatively unexplored NLP task called Semantic Overlap Summarization (SOS), which entails generating a single summary from multiple alternative narratives which can convey the…
SentenceText SummarizationOn the Persistent Effects of Lexicality in Large Language Models
Representations extracted from large language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than seman…
Semantic SimilarityBinary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models
Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of large language models (LLMs) for binary code compreh…
BenchmarkingCode SummarizationGPUSemantic Similarity+1Analyzing the Performance of Large Language Models on Code Summarization
Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We show that for the task of code summariza…
Code GenerationCode Summarization