Understanding Retrieval Augmentation for Long-Form Question Answering
We present a study of retrieval-augmented language models (LMs) on long-form question answering. We analyze how retrieval augmentation impacts different LMs, by comparing answers generated from models while using the same evidence documents, and how differing quality of retrieval document set impacts the answers generated from the same LM. We study various attributes of generated answers (e.g., fluency, length, variance) with an emphasis on the attribution of generated long-form answers to in-context evidence documents. We collect human annotations of answer attribution and evaluate methods for automatically judging attribution. Our study provides new insights on how retrieval augmentation impacts long, knowledge-rich text generation of LMs. We further identify attribution patterns for long text generation and analyze the main culprits of attribution errors. Together, our analysis reveals how retrieval augmentation impacts long knowledge-rich text generation and provide directions for future work.
Code (0)
등록된 구현이 없습니다.
Tasks
FormLong Form Question AnsweringQuestion AnsweringRetrievalText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Redefining Information Retrieval of Structured Database via Large Language Models
Retrieval augmentation is critical when Language Models (LMs) exploit non-parametric knowledge related to the query through external knowledge bases before reasoning. The retrieved information is incorporated into LMs as…
Information RetrievalQuestion AnsweringRetrievalMulti-Modal Retrieval Augmentation for Open-Ended and Knowledge-Intensive Video Question Answering
While current video question answering systems perform well on some tasks requiring only direct visual understanding, they struggle with questions demanding knowledge beyond what is immediately observable in the video co…
Multiple-choiceQuestion AnsweringRetrievalRetrieval-augmented Generation+2A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding
Users of medical question answering systems often submit long and detailed questions, making it hard to achieve high recall in answer retrieval. To alleviate this problem, we propose a novel Multi-Task Learning (MTL) met…
Data AugmentationDecoderMedical Question AnsweringMulti-Task Learning+2Retrieval meets Long Context Large Language Models
Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation v…
16k4kFew-Shot LearningNatural Questions+2On the Summarization of Consumer Health Questions
Question understanding is one of the main challenges in question answering. In real world applications, users often submit natural language questions that are longer than needed and include peripheral information that in…
Data AugmentationQuestion AnsweringRetrieval