TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
LLMs have achieved remarkable fluency and coherence in text generation, yet their widespread adoption has raised concerns about content reliability and accountability. In high-stakes domains such as healthcare, law, and news, it is crucial to understand where and how the content is created. To address this, we introduce the Text pROVEnance (TROVE) challenge, designed to trace each sentence of a target text back to specific source sentences within potentially lengthy or multi-document inputs. Beyond identifying sources, TROVE annotates the fine-grained relationships (quotation, compression, inference, and others), providing a deep understanding of how each target sentence is formed. To benchmark TROVE, we construct our dataset by leveraging three public datasets covering 11 diverse scenarios (e.g., QA and summarization) in English and Chinese, spanning source texts of varying lengths (0-5k, 5-10k, 10k+), emphasizing the multi-document and long-document settings essential for provenance. To ensure high-quality data, we employ a three-stage annotation process: sentence retrieval, GPT provenance, and human provenance. We evaluate 11 LLMs under direct prompting and retrieval-augmented paradigms, revealing that retrieval is essential for robust performance, larger models perform better in complex relationship classification, and closed-source models often lead, yet open-source models show significant promise, particularly with retrieval augmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSentenceSentence RetrievalText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GenProve: Learning to Generate Text with Fine-Grained Provenance
Large language models (LLM) often hallucinate, and while adding citations is a common solution, it is frequently insufficient for accountability as users struggle to verify how a cited source supports a generated claim. …
What is Your Article Based On? Inferring Fine-grained Provenance
When evaluating an article and the claims it makes, a critical reader must be able to assess where the information presented comes from, and whether the various claims are mutually consistent and support the conclusion. …
ArticlesFact CheckingSentenceFacilitating the sharing of electrophysiology data analysis results through in-depth provenance capture
Scientific research demands reproducibility and transparency, particularly in data-intensive fields like electrophysiology. Electrophysiology data is typically analyzed using scripts that generate output files, including…
Transfer LearningBileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature
Text watermarks for large language models (LLMs) have been commonly used to identify the origins of machine-generated content, which is promising for assessing liability when combating deepfake or harmful content. While …
Face SwappingImproving Neural Argumentative Stance Classification in Controversial Topics with Emotion-Lexicon Features
Argumentation mining comprises several subtasks, among which stance classification focuses on identifying the standpoint expressed in an argumentative text toward a specific target topic. While arguments-especially about…