MultiReQA: A Cross-Domain Evaluation forRetrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al.,2019).This dataset paper presents MultiReQA, a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets. We explore systematic retrieval based evaluation and transfer learning across domains over these datasets using a number of strong base-lines including two supervised neural models, based on fine-tuning BERT and USE-QA models respectively, as well as a surprisingly effective information retrieval baseline, BM25. Five of these tasks contain both training and test data, while three contain test data only. Performing cross training on the five tasks with training data shows that while a general model covering all domains is achievable, the best performance is often obtained by training exclusively on in-domain data.
Code (1)
Tasks
Information RetrievalQuestion AnsweringRetrievalSentenceTransfer LearningSimilar Papers 제목 키워드 기반
MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al.,2019).This paper presents MultiReQA, anew multi-domain ReQA evaluation suite com-posed…
Information RetrievalQuestion AnsweringRetrievalSentenceExploring Dual Encoder Architectures for Question Answering
Dual encoders have been used for question-answering (QA) and information retrieval (IR) tasks with good results. Previous research focuses on two major types of dual encoders, Siamese Dual Encoder (SDE), with parameters …
Information RetrievalQuestion AnsweringRetrievalSamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives
Dual encoders have been used for retrieval tasks and representation learning with good results. A standard way to train dual encoders is using a contrastive loss with in-batch negatives. In this work, we propose an impro…
Contrastive LearningInformation RetrievalQuestion AnsweringRepresentation Learning+1MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set compri…
Extractive Question-AnsweringNatural QuestionsOpen-Domain Question AnsweringQuestion Answering+1MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark
With the rapid advancement of Multimodal Large Language Models (MLLMs), numerous evaluation benchmarks have emerged. However, comprehensive assessments of their performance across diverse industrial applications remain l…
MMEModel OptimizationOptical Character Recognition (OCR)