CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
Code (5)
Tasks
DiagnosticQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Visual ReasoningSimilar Papers 제목 키워드 기반
QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning
Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR (johnson2017clevr), for example, tests a range of visual reasoning abilities. The questions …
DiagnosticQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)+1CLEAR: A Dataset for Compositional Language and Elementary Acoustic Reasoning
We introduce the task of acoustic question answering (AQA) in the area of acoustic reasoning. In this task an agent learns to answer questions on the basis of acoustic context. In order to promote research in this area, …
Acoustic Question AnsweringQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)ClevrSkills: Compositional Language and Visual Reasoning in Robotics
Robotics tasks are highly compositional by nature. For example, to perform a high-level task like cleaning the table a robot must employ low-level capabilities of moving the effectors to the objects on the table, pick th…
Visual ReasoningCLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image, using the conversation history as context. It entails challenges in vision, language, reasoning, and grounding. However, study…
coreference-resolutionCoreference ResolutionDiagnosticVisual DialogIterated learning for emergent systematicity in VQA
Although neural module networks have an architectural bias towards compositionality, they require gold standard layouts to generalize systematically in practice. When instead learning layouts and modules jointly, composi…
Question AnsweringSystematic GeneralizationVisual Question AnsweringVisual Question Answering (VQA)