Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation
The use of large language models (LLMs) has significantly increased since the introduction of ChatGPT in 2022, demonstrating their value across various applications. However, a major challenge for enterprise and commercial adoption of LLMs is their tendency to generate inaccurate information, a phenomenon known as "hallucination." This project proposes a method for estimating the factuality of a summary generated by LLMs when compared to a source text. Our approach utilizes Naive Bayes classification to assess the accuracy of the content produced.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationRetrieval-augmented GenerationText SummarizationSimilar Papers 제목 키워드 기반
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-g…
ArticlesNatural Language InferenceSummarization Consistency EvaluationText SummarizationOpenFActScore: Open-Source Atomic Evaluation of Factuality in Text Generation
We introduce OpenFActScore, an open-source implementation of the FActScore framework for evaluating the factuality of text generated by large language models (LLMs). FActScore evaluates the factual accuracy of long-form …
Text GenerationEvaluation of Automatic Text Summarization using Synthetic Facts
Despite some recent advances, automatic text summarization remains unreliable, elusive, and of limited practical use in applications. Two main problems with current summarization methods are well known: evaluation and fa…
Text SummarizationFactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Evaluating the factual consistency of automatically generated summaries is essential for the progress and adoption of reliable summarization systems. Despite recent advances, existing factuality evaluation models are not…
News SummarizationMolecular Facts: Desiderata for Decontextualization in LLM Fact Verification
Automatic factuality verification of large language model (LLM) generations is becoming more and more widely used to combat hallucinations. A major point of tension in the literature is the granularity of this fact-check…
Fact CheckingFact VerificationLanguage ModelingLanguage Modelling+1