Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding
Improving the quality of model-generated summaries, especially factuality, the accuracy of a summary with respect to its source content, remains a challenge. While reranking could select the optimal output from multiple generated candidates, it is limited to only using the source as guidance, resulting in unreliable summaries. To address this limitation, we propose ConSUM that reranks candidate summaries by considering two factors: consistency to the source document and consensus among the other candidates. Consensus is established using Minimum Bayes Risk (MBR) decoding over the set of generated summaries, while ensuring consistency by employing factuality-aware metrics that compare the summary against the source. Rigorous testing demonstrates that our system is competitive with existing methods, with human evaluations further confirming that its generated summaries are preferred over those from other systems. Our code is available at https://github.com/naist-nlp/ConSUM .
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation
Ensuring factual consistency is crucial for natural language generation tasks, particularly in abstractive summarization, where preserving the integrity of information is paramount. Prior works on evaluating factual cons…
Abstractive Text SummarizationNatural Language InferenceText GenerationFENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-g…
ArticlesNatural Language InferenceSummarization Consistency EvaluationText SummarizationFactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Evaluating the factual consistency of automatically generated summaries is essential for the progress and adoption of reliable summarization systems. Despite recent advances, existing factuality evaluation models are not…
News SummarizationImproving Factuality of Abstractive Summarization without Sacrificing Summary Quality
Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality.…
Abstractive Text SummarizationContrastive LearningGO FIGURE: A Meta Evaluation of Factuality in Summarization
While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question. This major discrepancy between the surface-level …
Common Sense ReasoningDiagnosticDocument SummarizationText Generation