Question Answering Infused Pre-training of General-Purpose Contextualized Representations
We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context. To this end, we train a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs. By encoding QA-relevant information, the bi-encoder's token-level representations are useful for non-QA downstream tasks without extensive (or in some cases, any) fine-tuning. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection on four datasets, few-shot named entity recognition on two datasets, and zero-shot sentiment analysis on three datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Question AnsweringSentiment AnalysisSimilar Papers 제목 키워드 기반
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Question Answering+1Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks
This paper explores domain adaptation for enabling question answering (QA) systems to answer questions posed against documents in new specialized domains. Current QA systems using deep neural network (DNN) technology hav…
Domain AdaptationQuestion AnsweringTransfer LearningPrecision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
In the realm of multimodal tasks, Visual Question Answering (VQA) plays a crucial role by addressing natural language questions grounded in visual content. Knowledge-Based Visual Question Answering (KBVQA) advances this …
DecoderKnowledge GraphsLanguage ModellingQuestion Answering+2Adapting general-purpose speech recognition engine output for domain-specific natural language question answering
Speech-based natural language question-answering interfaces to enterprise systems are gaining a lot of attention. General-purpose speech engines can be integrated with NLP systems to provide such interfaces. Usually, gen…
Question AnsweringSentencespeech-recognitionSpeech RecognitionKIQA: Knowledge-Infused Question Answering Model for Financial Table-Text Data
While entity retrieval models continue to advance their capabilities, our understanding of their wide-ranging applications is limited, especially in domain-specific settings. We highlighted this issue by using recent gen…
Entity LinkingEntity RetrievalLanguage ModelingLanguage Modelling+2