Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work, we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency and explore the ensuing trade-offs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationNatural Language Inferencereinforcement-learningReinforcement LearningText GenerationSimilar Papers 제목 키워드 기반
Evaluating Factuality in Generation with Dependency-level Entailment
Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailm…
ARCNatural Language InferenceQuestion GenerationQuestion-Generation+2AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation
Ensuring factual consistency is crucial for natural language generation tasks, particularly in abstractive summarization, where preserving the integrity of information is paramount. Prior works on evaluating factual cons…
Abstractive Text SummarizationNatural Language InferenceText GenerationEvaluating the Factual Consistency of Large Language Models Through News Summarization
While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its inpu…
News SummarizationCLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive t…
Abstractive Text SummarizationArticlesContrastive LearningRerankingMasked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking
Despite the recent advances in abstractive summarization systems, it is still difficult to determine whether a generated summary is factual consistent with the source text. To this end, the latest approach is to train a …
Abstractive Text Summarization