Leveraging Visual Question Answering for Image-Caption Ranking
Visual Question Answering (VQA) is the task of taking as input an image and a free-form natural language question about the image, and producing an accurate answer. In this work we view VQA as a "feature extraction" module to extract image and caption representations. We employ these representations for the task of image-caption ranking. Each feature dimension captures (imagines) whether a fact (question-answer pair) could plausibly be true for the image and caption. This allows the model to interpret images and captions from a wide variety of perspectives. We propose score-level and representation-level fusion models to incorporate VQA knowledge in an existing state-of-the-art VQA-agnostic image-caption ranking model. We find that incorporating and reasoning about consistency between images and captions significantly improves performance. Concretely, our model improves state-of-the-art on caption retrieval by 7.1% and on image retrieval by 4.4% on the MSCOCO dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Image RetrievalQuestion AnsweringRetrievalVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
Visual question answering (VQA) is known as an AI-complete task as it requires understanding, reasoning, and inferring about the vision and the language content. Over the past few years, numerous neural architectures hav…
Image CaptioningQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Generating Question Relevant Captions to Aid Visual Question Answering
Visual question answering (VQA) and image captioning require a shared body of general knowledge connecting language and vision. We present a novel approach to improve VQA performance that exploits this connection by join…
General KnowledgeImage CaptioningQuestion AnsweringVisual Question Answering+1Joint Image Captioning and Question Answering
Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images, which is too difficult for VQA systems to learn from images with the only supervision from…
Image CaptioningQuestion AnsweringVisual Question Answering (VQA)Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
Much recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-l…
General KnowledgeImage CaptioningQuestion AnsweringVisual Question Answering+1Improving Visual Question Answering by Referring to Generated Paragraph Captions
Paragraph-style image captions describe diverse aspects of an image as opposed to the more common single-sentence captions that only provide an abstract description of the image. These paragraph captions can hence contai…
DecoderImage CaptioningQuestion AnsweringSentence+2