Visual Question: Predicting If a Crowd Will Agree on the Answer
Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response. However, close examination of the VQA problem reveals an unavoidable, entangled problem that multiple humans may or may not always agree on a single answer to a visual question. We train a model to automatically predict from a visual question whether a crowd would agree on a single answer. We then propose how to exploit this system in a novel application to efficiently allocate human effort to collect answers to visual questions. Specifically, we propose a crowdsourcing system that automatically solicits fewer human responses when answer agreement is expected and more human responses when answer disagreement is expected. Our system improves upon existing crowdsourcing systems, typically eliminating at least 20% of human effort with no loss to the information collected from the crowd.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringvalidVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data
We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously…
Agree or Disagree: Predicting Judgments on Nuanced Assertions
Being able to predict whether people agree or disagree with an assertion (i.e. an explicit, self-contained statement) has several applications ranging from predicting how many people will like or dislike a social media p…
Learning Ambiguity from Crowd Sequential Annotations
Most crowdsourcing learning methods treat disagreement between annotators as noisy labelings while inter-disagreement among experts is often a good indicator for the ambiguity and uncertainty that is inherent in natural …
NERPOSPOS TaggingAmbiguity Helps: Classification With Disagreements in Crowdsourced Annotations
Imagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to…
ClassificationGeneral ClassificationCrowd disagreement about medical images is informative
Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing…
Medical Image Analysis