Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks
Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks such as open question answering and fact verification. These models are trained to generate the final output given the retrieved passages, which can be irrelevant to the original query, leading to learning spurious cues or answer memorization. This work introduces a method to incorporate the evidentiality of passages -- whether a passage contains correct evidence to support the output -- into training the generator. We introduce a multi-task learning framework to jointly generate the final output and predict the evidentiality of each passage, leveraging a new task-agnostic method to obtain silver evidentiality labels for supervision. Our experiments on five datasets across three knowledge-intensive tasks show that our new evidentiality-guided generator significantly outperforms its direct counterpart with the same-size model and advances the state of the art on FaVIQ-Ambig. We attribute these improvements to both the auxiliary multi-task learning and silver evidentiality mining techniques.
Code (1)
Tasks
AttributeFact VerificationMemorizationMulti-Task LearningOpen-Ended Question AnsweringQuestion AnsweringRetrievalRetrieval-augmented GenerationSimilar Papers 제목 키워드 기반
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks
Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks such as open question answering and fact verification. These models are trained to generate a final …
Fact VerificationMemorizationMulti-Task LearningOpen-Domain Question Answering+4ECoRAG: Evidentiality-guided Compression for Long Context RAG
Large Language Models (LLMs) have shown remarkable performance in Open-Domain Question Answering (ODQA) by leveraging external documents through Retrieval-Augmented Generation (RAG). To reduce RAG overhead, from longer c…
Answer GenerationOpen-Domain Question AnsweringQuestion AnsweringRAG+1Robustifying Multi-hop QA through Pseudo-Evidentiality Training
This paper studies the bias problem of multi-hop question answering models, of answering correctly without correct reasoning. One way to robustify these models is by supervising to not only answer right, but also with ri…
counterfactualMulti-hop Question AnsweringQuestion AnsweringWhen Evidence Shapes Collaboration: Knowledge-Conditioned Topology Generation for Multi-Agent Systems
Multi-Agent Systems (MAS) have recently moved from static workflows toward dynamically generated collaboration topologies. However, existing topology generation methods rely primarily on the parametric knowledge of large…
Graph GenerationEvidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering
The long-standing goal of dense retrievers in abtractive open-domain question answering (ODQA) tasks is to learn to capture evidence passages among relevant passages for any given query, such that the reader produce fact…
Contrastive LearningcounterfactualOpen-Domain Question AnsweringPassage Retrieval+3