Frustratingly Hard Evidence Retrieval for QA Over Books
A lot of progress has been made to improve question answering (QA) in recent years, but the special problem of QA over narrative book stories has not been explored in-depth. We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We further investigate how state-of-the-art open-domain QA approaches can help BookQA. Besides achieving state-of-the-art on the NarrativeQA benchmark, our study also reveals the difficulty of evidence retrieval in books with a wealth of experiments and analysis - which necessitates future effort on novel solutions for evidence retrieval in BookQA.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringRetrievalSimilar Papers 제목 키워드 기반
Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks
Retrieval-augmented Generation (RAG) has primarily been studied in limited settings, such as factoid question answering; more challenging, reasoning-intensive benchmarks have seen limited success from minimal RAG. In thi…
Question AnsweringMemorization: A Close Look at Books
To what extent can entire books be extracted from LLMs? Using the Llama 3 70B family of models, and the "prefix-prompting" extraction technique, we were able to auto-regressively reconstruct, with a very high level of si…
MemorizationFrustratingly Easy Uncertainty Estimation for Distribution Shift
Distribution shift is an important concern in deep image classification, produced either by corruption of the source images, or a complete change, with the solution involving domain adaptation. While the primary goal is …
Domain Adaptationimage-classificationImage ClassificationUnsupervised Domain AdaptationIllocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
Natural language explanations produced by large language models (LLMs) are often persuasive, but not necessarily scrutable: users cannot easily verify whether the claims in an explanation are supported by evidence. In XA…
LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval
Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and support…