Crowdsourcing Multiple Choice Science Questions
We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions (Dataset available at http://allenai.org/data.html). We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityMultiple-choiceQuestion GenerationQuestion-GenerationSimilar Papers 제목 키워드 기반
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC has been influential in the community, its multiple-choice form…
AI2 Reasoning ChallengeARCMultiple-choiceNatural Questions+2From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy, but the rich variety of standardized exams has remained a landmark challenge. Even in 2016, the best AI system achieved merel…
Multiple-choiceQuestion AnsweringLooking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. We solicit and verify questions and answers for this challenge through a 4-…
DiversityNatural Language InferenceQuestion AnsweringReading Comprehension+1TabMCQ: A Dataset of General Knowledge Tables and Multiple-choice Questions
We describe two new related resources that facilitate modelling of general knowledge reasoning in 4th grade science exams. The first is a collection of curated facts in the form of tables, and the second is a large set o…
General KnowledgeMultiple-choiceQuestion AnsweringHypothesis Testing for Quantifying LLM-Human Misalignment in Multiple Choice Settings
As Large Language Models (LLMs) increasingly appear in social science research (e.g., economics and marketing), it becomes crucial to assess how well these models replicate human behavior. In this work, using hypothesis …
Decision MakingLanguage ModelingLanguage ModellingMarketing+1