Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions
Modern question answering systems have been touted as approaching human performance. However, existing question answering datasets are imperfect tests. Questions are written with humans in mind, not computers, and often do not properly expose model limitations. To address this, we develop an adversarial writing setting, where humans interact with trained models and try to break them. This annotation process yields a challenge set, which despite being easy for trivia players to answer, systematically stumps automated question answering systems. Diagnosing model errors on the evaluation data provides actionable insights to explore in developing robust and generalizable question answering systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
A novel interface for adversarial trivia question-writing
A critical component when developing question-answering AIs is an adversarial dataset that challenges models to adapt to the complex syntax and reasoning underlying our natural language. Present techniques for procedural…
Question AnsweringSentenceTrick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering
Adversarial evaluation stress tests a model's understanding of natural language. While past approaches expose superficial patterns, the resulting adversarial examples are limited in complexity and diversity. We propose h…
DiversityInformation RetrievalQuestion AnsweringRetrievalHow the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation
Dynamic adversarial question generation, where humans write examples to stump a model, aims to create examples that are realistic and informative. However, the advent of large language models (LLMs) has been a double-edg…
Question GenerationQuestion-GenerationRetrievalControllable Decontextualization of Yes/No Question and Answers into Factual Statements
Yes/No or polar questions represent one of the main linguistic question categories. They consist of a main interrogative clause, for which the answer is binary (assertion or negation). Polar questions and answers (PQA) r…
NegationTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathere…
Reading ComprehensionSentenceTriviaQA