Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization
Existing factual consistency evaluation approaches for text summarization provide binary predictions and limited insights into the weakness of summarization systems. Therefore, we propose the task of fine-grained inconsistency detection, the goal of which is to predict the fine-grained types of factual errors in a summary. Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact, which explicitly represents the facts in the documents and summaries with semantic frames extracted by semantic role labeling, and highlights the related semantic frames to predict inconsistency. The highlighted semantic frames help verify predicted error types and correct inconsistent summaries. Experiment results demonstrate that our model outperforms strong baselines and provides evidence to support or refute the summary.
Code (1)
Tasks
Semantic Role LabelingText SummarizationSimilar Papers 제목 키워드 기반
FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in a…
Abstractive Text SummarizationDQE: A Semantic-Aware Evaluation Metric for Time Series Anomaly Detection
Time series anomaly detection has achieved remarkable progress in recent years. However, evaluation practices have received comparatively less attention, despite their critical importance. Existing metrics exhibit severa…
Time Series Anomaly DetectionFG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning
Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as mathematical problem-solving. Existing approaches primarily detect the presence of hallu…
GSM8KHallucinationLanguage ModelingLanguage Modelling+3RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought
Large language Models (LLMs) have achieved promising performance on arithmetic reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting. However, LLMs face challenges in maintaining factual consiste…
Arithmetic ReasoningGSM8KHallucinationNeural models for Factual Inconsistency Classification with Explanations
Factual consistency is one of the most important requirements when editing high quality documents. It is extremely important for automatic text generation systems like summarization, question answering, dialog modeling, …
8kClassificationLanguage ModelingLanguage Modelling+4