Passage Segmentation of Documents for Extractive Question Answering
Retrieval-Augmented Generation (RAG) has proven effective in open-domain question answering. However, the chunking process, which is essential to this pipeline, often receives insufficient attention relative to retrieval and synthesis components. This study emphasizes the critical role of chunking in improving the performance of both dense passage retrieval and the end-to-end RAG pipeline. We then introduce the Logits-Guided Multi-Granular Chunker (LGMGC), a novel framework that splits long documents into contextualized, self-contained chunks of varied granularity. Our experimental results, evaluated on two benchmark datasets, demonstrate that LGMGC not only improves the retrieval step but also outperforms existing chunking methods when integrated into a RAG pipeline.
Code (0)
등록된 구현이 없습니다.
Tasks
ChunkingExtractive Question-AnsweringOpen-Domain Question AnsweringPassage RetrievalQuestion AnsweringRAGRetrievalRetrieval-augmented GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AutoEQA: Auto-Encoding Questions for Extractive Question Answering
There has been a significant progress in the field of Extractive Question Answering (EQA) in the recent years. However, most of them are reliant on annotations of answer-spans in the corresponding passages. In this work,…
Extractive Question-AnsweringQuestion AnsweringQuestion GenerationQuestion-GenerationA Method of Passage-Based Document Retrieval in Question Answering System
We propose a method for using the scoring values of passages to effectively retrieve documents in a Question Answering system. For this, we suggest evaluation function that considers proximity between each question ter…
Question AnsweringRetrievalConversational Answer Generation and Factuality for Reading Comprehension Question-Answering
Question answering (QA) is an important use case on voice assistants. A popular approach to QA is extractive reading comprehension (RC) which finds an answer span in a text passage. However, extractive answers are often …
Answer GenerationPassage RankingQuestion AnsweringReading ComprehensionDelaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering
Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability …
Information RetrievalManagementOpen-Domain Question AnsweringQuestion Answering+1Attention-guided Generative Models for Extractive Question Answering
We propose a novel method for applying Transformer models to extractive question answering (QA) tasks. Recently, pretrained generative sequence-to-sequence (seq2seq) models have achieved great success in question answeri…
DecoderExtractive Question-AnsweringHallucinationOpen-Domain Question Answering+2