Measuring Machine Intelligence Through Visual Question Answering
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine's ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Measuring CLEVRness: Black-box Testing of Visual Reasoning Models
How to measure the reasoning capabilities of intelligence systems? Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the scene.…
BenchmarkingDiagnosticQuestion AnsweringVisual Question Answering+2Measuring CLEVRness: Blackbox testing of Visual Reasoning Models
How can we measure the reasoning capabilities of intelligence systems? Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the sc…
BenchmarkingDiagnosticQuestion AnsweringVisual Question Answering+2RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
Dramatic progress has been witnessed in basic vision tasks involving low-level perception, such as object recognition, detection, and tracking. Unfortunately, there is still an enormous performance gap between artificial…
Object RecognitionQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)+1What Images are More Memorable to Machines?
This paper studies the problem of measuring and predicting how memorable an image is to pattern recognition machines, as a path to explore machine intelligence. Firstly, we propose a self-supervised machine memory quanti…
Measuring AI Systems Beyond Accuracy
Current test and evaluation (T&E) methods for assessing machine learning (ML) system performance often rely on incomplete metrics. Testing is additionally often siloed from the other phases of the ML system lifecycle. Re…