What Question Answering can Learn from Trivia Nerds
In addition to the traditional task of getting machines to answer questions, a major research question in question answering is to create interesting, challenging questions that can help systems learn how to answer questions and also reveal which systems are the best at answering questions. We argue that creating a question answering dataset -- and the ubiquitous leaderboard that goes with it -- closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the decades of hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons -- removing ambiguity, discriminating skill, and adjudicating disputes -- that can transfer to QA research and how they might be implemented for the QA community.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
Quizbowl: The Case for Incremental Question Answering
Scholastic trivia competitions test knowledge and intelligence through mastery of question answering. Modern question answering benchmarks are one variant of the Turing test. Specifically, answering a set of questions as…
BIG-bench Machine LearningDecision MakingQuestion AnsweringSequential Decision MakingThe meaning of "most" for visual question answering models
The correct interpretation of quantifier statements in the context of a visual scene requires non-trivial inference mechanisms. For the example of "most", we discuss two strategies which rely on fundamentally different c…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)The Meaning of ``Most'' for Visual Question Answering Models
The correct interpretation of quantifier statements in the context of a visual scene requires non-trivial inference mechanisms. For the example of {``}most{''}, we discuss two strategies which rely on fundamentally diffe…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions
Modern question answering systems have been touted as approaching human performance. However, existing question answering datasets are imperfect tests. Questions are written with humans in mind, not computers, and often …
Question AnsweringResearchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ``known unknown…
Known UnknownsQuestion AnsweringTriviaQA