SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
Ensuring factual consistency between the summary and the original document is paramount in summarization tasks. Consequently, considerable effort has been dedicated to detecting inconsistencies. With the advent of Large Language Models (LLMs), recent studies have begun to leverage their advanced language understanding capabilities for inconsistency detection. However, early attempts have shown that LLMs underperform traditional models due to their limited ability to follow instructions and the absence of an effective detection methodology. In this study, we reassess summary inconsistency detection with LLMs, comparing the performances of GPT-3.5 and GPT-4. To advance research in LLM-based inconsistency detection, we propose SIFiD (Summary Inconsistency Detection with Filtered Document) that identify key sentences within documents by either employing natural language inference or measuring semantic similarity between summaries and documents.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceSemantic SimilaritySemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in a…
Abstractive Text SummarizationInterpretable Automatic Fine-grained Inconsistency Detection in Text Summarization
Existing factual consistency evaluation approaches for text summarization provide binary predictions and limited insights into the weakness of summarization systems. Therefore, we propose the task of fine-grained inconsi…
Semantic Role LabelingText SummarizationCoP: Factual Inconsistency Detection by Controlling the Preference
Abstractive summarization is the process of generating a summary given a document as input. Although significant progress has been made, the factual inconsistency between the document and the generated summary still limi…
Abstractive Text SummarizationDialogue Inspectional Summarization with Factual Inconsistency Awareness
Dialogue summarization has been extensively studied and applied, where the prior works mainly focused on exploring superior model structures to align the input dialogue and the output summary. However, for professional d…
dialogue summaryMedical DiagnosisSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when a…
Natural Language InferenceSentence