FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents
We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents.
Code (1)
Tasks
Claim VerificationRAGMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ExClaim: Explainable Neural Claim Verification Using Rationalization
With the advent of deep learning, text generation language models have improved dramatically, with text at a similar level as human-written text. This can lead to rampant misinformation because content can now be created…
Claim VerificationDecision MakingExplainable Artificial Intelligence (XAI)Misinformation+1DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification
Recently, many methods discover effective evidence from reliable sources by appropriate neural networks for explainable claim verification, which has been widely recognized. However, in these methods, the discovery proce…
Claim VerificationEX-FEVER: A Dataset for Multi-hop Explainable Fact Verification
Fact verification aims to automatically probe the veracity of a claim based on several pieces of evidence. Existing works are always engaging in accuracy improvement, let alone explainability, a critical capability of fa…
Claim VerificationExplanation GenerationFact VerificationRetrieval+1MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval
Verifying the truthfulness of claims usually requires joint multi-modal reasoning over both textual and visual evidence, such as analyzing both textual caption and chart image for claim verification. In addition, to make…
Explanation GenerationStep-by-Step Fact Verification System for Medical Claims with Explainable Reasoning
Fact verification (FV) aims to assess the veracity of a claim based on relevant evidence. The traditional approach for automated FV includes a three-part pipeline relying on short evidence snippets and encoder-only infer…
Fact CheckingFact Verification