SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits
Detecting factual inconsistencies in summarization is critical, yet existing benchmarks lack the necessary challenge and interpretability for robust evaluation. In this paper, we introduce SummExecEdit, a novel benchmark leveraging executable edits to assess models on their ability to both detect factual errors and provide accurate explanations. The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark, with individual scores of 0.67 for detection and 0.73 for explanation. Furthermore, we identify four primary types of explanation errors, with 45.4% of errors focusing on completely unrelated parts of the summary.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive summarization systems. However, the formulation of the task is rather ambiguous, neither the l…
Abstractive Text SummarizationvalidEvaluating the Factual Consistency of Large Language Models Through News Summarization
While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its inpu…
News SummarizationDiscourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization
Detecting factual inconsistency for long document summarization remains challenging, given the complex structure of the source article and long summary length. In this work, we study factual inconsistency errors and conn…
Document SummarizationNatural Language InferenceSentenceConstrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation
Despite significant progress, state-of-the-art abstractive summarization methods are still prone to hallucinate content inconsistent with the source document. In this paper, we propose Constrained Abstractive Summarizati…
Abstractive Text SummarizationKeyphrase ExtractionMasked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking
Despite the recent advances in abstractive summarization systems, it is still difficult to determine whether a generated summary is factual consistent with the source text. To this end, the latest approach is to train a …
Abstractive Text Summarization