Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive summarization systems. However, the formulation of the task is rather ambiguous, neither the linguistic nor the natural language processing community has succeeded in giving a mutually agreed-upon definition. Due to this lack of well-defined formulation, a large number of popular abstractive summarization datasets are constructed in a manner that neither guarantees validity nor meets one of the most essential criteria of summarization: factual consistency. In this paper, we address this issue by combining state-of-the-art factual consistency models to identify the problematic instances present in popular summarization datasets. We release SummFC, a filtered summarization dataset with improved factual consistency, and demonstrate that models trained on this dataset achieve improved performance in nearly all quality aspects. We argue that our dataset should become a valid benchmark for developing and evaluating summarization systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationvalidSimilar Papers 제목 키워드 기반
A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarization
Although large language models (LLMs) have shown promising potential in news summarization tasks, their performance on long-document summarization remains challenging as their length often exceeds the input limits. As th…
Document SummarizationRobust Counterfactual Explanations for Tree-Based Ensembles
Counterfactual explanations inform ways to achieve a desired outcome from a machine learning model. However, such explanations are not robust to certain real-world changes in the underlying model (e.g., retraining the mo…
counterfactualCounterfactual Self-Questioning for Stable Policy Optimization in Language Models
Recent work on language model self-improvement shows that models can refine their own reasoning through reflection, verification, debate, or self-generated rewards. However, most existing approaches rely on external crit…
Mathematical ReasoningGCFX: Generative Counterfactual Explanations for Deep Graph Models at the Model Level
Deep graph learning models have demonstrated remarkable capabilities in processing graph-structured data and have been widely applied across various fields. However, their complex internal architectures and lack of trans…
Graph GenerationGraph LearningAnnotating and Modeling Fine-grained Factuality in Summarization
Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input a…
Abstractive Text SummarizationSentence