Revisiting text decomposition methods for NLI-based factuality scoring of summaries
Scoring the factuality of a generated summary involves measuring the degree to which a target text contains factual information using the input document as support. Given the similarities in the problem formulation, previous work has shown that Natural Language Inference models can be effectively repurposed to perform this task. As these models are trained to score entailment at a sentence level, several recent studies have shown that decomposing either the input document or the summary into sentences helps with factuality scoring. But is fine-grained decomposition always a winning strategy? In this paper we systematically compare different granularities of decomposition -- from document to sub-sentence level, and we show that the answer is no. Our results show that incorporating additional context can yield improvement, but that this does not necessarily apply to all datasets. We also show that small changes to previously proposed entailment-based scoring methods can result in better performance, highlighting the need for caution in model and methodology selection for downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceSentenceSimilar Papers 제목 키워드 기반
AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification
Despite the significant advancements of Large Language Models (LLMs), their factuality remains a critical challenge, creating a growing need for more nuanced factuality verification. Existing factuality verification meth…
Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMs
In recent years, Large Language Models (LLMs) have gained immense attention due to their notable emergent capabilities, surpassing those seen in earlier language models. A particularly intriguing application of LLMs is t…
BenchmarkingQuestion AnsweringText GenerationMedScore: Factuality Evaluation of Free-Form Medical Answers
While Large Language Models (LLMs) can generate fluent and convincing responses, they are not necessarily correct. This is especially apparent in the popular decompose-then-verify factuality evaluation pipeline, where LL…
FormHallucinationSentencevalidDnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation
The decompose-then-verify strategy for verification of Large Language Model (LLM) generations decomposes claims that are then independently verified. Decontextualization augments text (claims) to ensure it can be verifie…
FormLanguage ModelingLanguage ModellingLarge Language Model+1Neural models of factuality
We present two neural models for event factuality prediction, which yield significant performance gains over previous models on three event factuality datasets: FactBank, UW, and MEANTIME. We also present a substantial e…