Solving Visual Madlibs with Multiple Cues
This paper focuses on answering fill-in-the-blank style multiple choice questions from the Visual Madlibs dataset. Previous approaches to Visual Question Answering (VQA) have mainly used generic image features from networks trained on the ImageNet dataset, despite the wide scope of questions. In contrast, our approach employs features derived from networks trained for specialized tasks of scene classification, person activity prediction, and person and object attribute prediction. We also present a method for selecting sub-regions of an image that are relevant for evaluating the appropriateness of a putative answer. Visual features are computed both from the whole image and from local regions, while sentences are mapped to a common space using a simple normalized canonical correlation analysis (CCA) model. Our results show a significant improvement over the previous state of the art, and indicate that answering different question types benefits from examining a variety of image cues and carefully choosing informative image sub-regions.
Code (0)
등록된 구현이 없습니다.
Tasks
Activity PredictionAttributeMultiple-choiceQuestion AnsweringScene ClassificationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Combining Multiple Cues for Visual Madlibs Question Answering
This paper presents an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset. Instead of generic and commonly used representations trained on the ImageNet classification task,…
AttributeGeneral ClassificationMultiple-choiceQuestion Answering+1Visual Madlibs: Fill in the blank Image Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-bla…
Image GenerationMultiple-choiceQuestion AnsweringVisual Madlibs: Fill in the Blank Description Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-bl…
Multiple-choiceQuestion AnsweringMean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
We present Mean Box Pooling, a novel visual representation that pools over CNN representations of a large number, highly overlapping object proposals. We show that such representation together with nCCA, a successful mul…
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
This paper proposes deep convolutional network models that utilize local and global context to make human activity label predictions in still images, achieving state-of-the-art performance on two recent datasets with hun…
General ClassificationHuman-Object Interaction DetectionMultiple-choiceMultiple Instance Learning+3