Context-Aware Transformer Pre-Training for Answer Sentence Selection
Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Context-based Transformer Models for Answer Sentence Selection
An important task for the design of Question Answering systems is the selection of the sentence containing (or constituting) the answer from documents relevant to the asked question. Most previous work has only used the …
Question AnsweringSentenceContextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task
Word embeddings that consider context have attracted great attention for various natural language processing tasks in recent years. In this paper, we utilize contextualized word embeddings with the transformer encoder fo…
Answer SelectionSentenceSentence SimilarityWord EmbeddingsLooComp: Leverage Leave-One-Out Strategy to Encoder-only Transformer for Efficient Query-aware Context Compression
Efficient context compression is crucial for improving the accuracy and scalability of question answering. For the efficiency of Retrieval Augmented Generation, context should be delivered fast, compact, and precise to e…
Question AnsweringTime-Aware Representation Learning for Time-Sensitive Question Answering
Time is one of the crucial factors in real-world question answering (QA) problems. However, language models have difficulty understanding the relationships between time specifiers, such as 'after' and 'before', and numbe…
Question AnsweringRepresentation LearningSentenceQuestion-Context Alignment and Answer-Context Dependencies for Effective Answer Sentence Selection
Answer sentence selection (AS2) in open-domain question answering finds answer for a question by ranking candidate sentences extracted from web documents. Recent work exploits answer context, i.e., sentences around a can…
Open-Domain Question AnsweringQuestion AnsweringSentence