Towards a Framework for Evaluating Explanations in Automated Fact Verification
As deep neural models in NLP become more complex, and as a consequence opaque, the necessity to interpret them becomes greater. A burgeoning interest has emerged in rationalizing explanations to provide short and coherent justifications for predictions. In this position paper, we advocate for a formal framework for key concepts and properties about rationalizing explanations to support their evaluation systematically. We also outline one such formal framework, tailored to rationalizing explanations of increasingly complex structures, from free-form explanations to deductive explanations, to argumentative explanations (with the richest structure). Focusing on the automated fact verification task, we provide illustrations of the use and usefulness of our formalization for evaluating explanations, tailored to their varying structures.
Code (1)
Tasks
Fact VerificationPositionSimilar Papers 제목 키워드 기반
Evaluating Evidence Attribution in Generated Fact Checking Explanations
Automated fact-checking systems often struggle with trustworthiness, as their generated explanations can include hallucinations. In this work, we explore evidence attribution for fact-checking explanation generation. We …
Explanation GenerationFact CheckingSynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification
Large Language Models (LLMs) with extended context windows promise direct reasoning over long documents, reducing the need for chunking or retrieval. Constructing annotated resources for training and evaluation, however,…
Evaluating Step-by-Step Reasoning through Symbolic Verification
Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be mo…
In-Context LearningFactLens: Benchmarking Fine-Grained Fact Verification
Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. To verify LL…
BenchmarkingFact VerificationText GenerationLogical Satisfiability of Counterfactuals for Faithful Explanations in NLI
Evaluating an explanation's faithfulness is desired for many reasons such as trust, interpretability and diagnosing the sources of model's errors. In this work, which focuses on the NLI task, we introduce the methodology…
counterfactual