CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
With the rise of large-scale pre-trained language models, open-domain question-answering (ODQA) has become an important research topic in NLP. Based on the popular pre-training fine-tuning approach, we posit that an additional in-domain pre-training stage using a large-scale, natural, and diverse question-answering (QA) dataset can be beneficial for ODQA. Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper. Using the readily available schema.org annotation, we extract around 130 million multilingual question-answer pairs, including about 60 million English data-points. With this previously unseen number of natural QA pairs, we pre-train popular language models to show the potential of large-scale in-domain pre-training for the task of question-answering. In our experiments, we find that pre-training question-answering models on our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks.
Code (1)
Tasks
Open-Domain Question AnsweringQuestion AnsweringSimilar Papers 제목 키워드 기반
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
We propose a novel open-domain question-answering dataset based on the Common Crawl project. With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English da…
Open-Domain Question AnsweringQuestion AnsweringAligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
Code Community Question Answering (CCQA) seeks to tackle programming-related issues, thereby boosting productivity in both software engineering and academic research. Recent advancements in Reinforcement Learning from Hu…
Community Question AnsweringIn-Context LearningQuestion AnsweringCCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMs
Recently, inference-time reasoning strategies have further improved the accuracy of large language models (LLMs), but their effectiveness on smaller models remains unclear. Based on the observation that conventional appr…
Question GenerationAutomating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models
Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering …
Information RetrievalQuestion AnsweringText GenerationHow to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering
Using deep learning models on small scale datasets would result in overfitting. To overcome this problem, the process of pre-training a model and fine-tuning it to the small scale dataset has been used extensively in dom…
Open-Domain Question AnsweringQuestion AnsweringReading Comprehension