Reducing Quantity Hallucinations in Abstractive Summarization
It is well-known that abstractive summaries are subject to hallucination---including material that is not supported by the original text. While summaries can be made hallucination-free by limiting them to general phrases, such summaries would fail to be very informative. Alternatively, one can try to avoid hallucinations by verifying that any specific entities in the summary appear in the original text in a similar context. This is the approach taken by our system, Herman. The system learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.) in a beam-worth of abstractive summaries produced by state-of-the-art models, in order to up-rank those summaries whose quantity terms are supported by the original text. Experimental results demonstrate that the ROUGE scores of such up-ranked summaries have a higher Precision than summaries that have not been up-ranked, without a comparable loss in Recall, resulting in higher F$_1$. Preliminary human evaluation of up-ranked vs. original summaries shows people's preference for the former.
Code (1)
Tasks
Abstractive Text SummarizationHallucinationSimilar Papers 제목 키워드 기반
Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search
Abstractive summarization systems today produce fluent and relevant output, but often "hallucinate" statements not supported by the source text. We analyze the connection between hallucinations and training data, and fin…
Abstractive Text SummarizationHallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text. Despite being assumed to be incorrect, we find that much hallucinated c…
Abstractive Text SummarizationReinforcement Learning (RL)World KnowledgeInvestigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Despite the remarkable performance of generative large language models (LLMs) on abstractive summarization, they face two significant challenges: their considerable size and tendency to hallucinate. Hallucinations are co…
Abstractive Text SummarizationHallucinationHallucination EvaluationApplicability of Large Language Models and Generative Models for Legal Case Judgement Summarization
Automatic summarization of legal case judgements, which are known to be long and complex, has traditionally been tried via extractive summarization models. In recent years, generative models including abstractive summari…
Abstractive Text SummarizationExtractive SummarizationHallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
State-of-the-art abstractive summarization systems often generate \emph{hallucinations}; i.e., content that is not directly inferable from the source text. Despite being assumed incorrect, we find that much hallucinated …
Abstractive Text SummarizationReinforcement Learning (RL)World Knowledge