Long Context Question Answering via Supervised Contrastive Learning
Long-context question answering (QA) tasks require reasoning over a long document or multiple documents. Addressing these tasks often benefits from identifying a set of evidence spans (e.g., sentences), which provide supporting evidence for answering the question. In this work, we propose a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence. We achieve this via an additional contrastive supervision signal in finetuning, where the model is encouraged to explicitly discriminate supporting evidence sentences from negative ones by maximizing question-evidence similarity. The proposed additional loss exhibits consistent improvements on three different strong long-context transformer models, across two challenging question answering benchmarks -- HotpotQA and QAsper.
Code (2)
Tasks
Contrastive LearningQuestion AnsweringVideo GenerationSimilar Papers 제목 키워드 기반
elBERto: Self-supervised Commonsense Learning for Question Answering
Commonsense question answering requires reasoning about everyday situations and causes and effects implicit in context. Typically, existing approaches first retrieve external evidence and then perform commonsense reasoni…
Question AnsweringRepresentation LearningRetrievalMuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads
Large Language Models (LLMs) frequently show distracted attention due to irrelevant information in the input, which severely impairs their long-context capabilities. Inspired by recent studies on the effectiveness of ret…
Contrastive LearningQuestion AnsweringRetrievalQA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation
Question answering (QA) has recently shown impressive results for answering questions from customized domains. Yet, a common challenge is to adapt QA models to an unseen target domain. In this paper, we propose a novel s…
Contrastive LearningData AugmentationDomain AdaptationQuestion AnsweringQAID: Question Answering Inspired Few-shot Intent Detection
Intent detection with semantically similar fine-grained intents is a challenging task. To address it, we reformulate intent detection as a question-answering retrieval task by treating utterances and intent names as ques…
Intent DetectionQuestion AnsweringRetrievalMomentum Contrastive Pre-training for Question Answering
Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In…
BenchmarkingContrastive LearningExtractive Question-AnsweringNatural Questions+1