Measuring Factual Consistency of Abstractive Summaries
Recent abstractive summarization systems fail to generate factual consistent -- faithful -- summaries, which heavily limits their practical application. Commonly, these models tend to mix concepts from the source or hallucinate new content, completely ignoring the source. Addressing the faithfulness problem is perhaps the most critical challenge for current abstractive summarization systems. First automatic faithfulness metrics were proposed, but we argue that existing methods do not yet utilize all "machinery" that this field has to offer and introduce new approaches to assess factual correctness. We evaluate existing and our proposed methods by correlating them with human judgements and find that BERTScore works well. Next, we conduct a data analysis, which reveals common problems, ways to further improve the metrics and indicates that combining multiple metrics is promising. Finally, we exploit faithfulness metrics in pre- and post-processing steps to decrease factual errors made by state-of-the-art summarization systems. We find that simple techniques like filtering training data and re-ranking generated summaries can increase the faithfulness by a substantial margin.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationRe-RankingSimilar Papers 제목 키워드 기반
Enhancing Factual Consistency of Abstractive Summarization
Automatic abstractive summaries are found to often distort or fabricate facts in the article. This inconsistency between summary and original text has seriously impacted its applicability. We propose a fact-aware summari…
Abstractive Text SummarizationGraph AttentionMulti-Fact Correction in Abstractive Text Summarization
Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated abstractive summaries often face the pitf…
Abstractive Text SummarizationNews SummarizationQuestion AnsweringText SummarizationImproving Factual Consistency of Abstractive Summarization via Question Answering
A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may pro…
Abstractive Text SummarizationQuestion AnsweringFactual Error Correction for Abstractive Summarization Models
Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency…
Abstractive Text SummarizationDo Automatic Factuality Metrics Measure Factuality? A Critical Evaluation
Modern LLMs can now produce highly readable abstractive summaries, to the point where traditional automated metrics for evaluating summary quality, such as ROUGE, have become saturated. However, LLMs still sometimes intr…