Generating Question Relevant Captions to Aid Visual Question Answering
Visual question answering (VQA) and image captioning require a shared body of general knowledge connecting language and vision. We present a novel approach to improve VQA performance that exploits this connection by jointly generating captions that are targeted to help answer a specific visual question. The model is trained using an existing caption dataset by automatically determining question-relevant captions using an online gradient-based method. Experimental results on the VQA v2 challenge demonstrates that our approach obtains state-of-the-art VQA performance (e.g. 68.4% on the Test-standard set using a single model) by simultaneously generating question-relevant captions.
Code (0)
등록된 구현이 없습니다.
Tasks
General KnowledgeImage CaptioningQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Multimodal Differential Network for Visual Question Generation
Generating natural questions from an image is a semantic task that requires using visual and language modality to learn multimodal representations. Images can have multiple visual and language contexts that are relevant …
Natural QuestionsQuestion GenerationQuestion-GenerationMultimodal Differential Network for Visual Question Generation
Generating natural questions from an image is a semantic task that requires using visual and language modality to learn multimodal representations. Images can have multiple visual and language contexts that are relevant …
Image CaptioningNatural QuestionsQuestion AnsweringQuestion Generation+2GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
Knowledge-based visual question answering (VQA) requires world knowledge beyond the image for accurate answer. Recently, instead of extra knowledge bases, a large language model (LLM) like GPT-3 is activated as an implic…
Language ModelingLanguage ModellingLarge Language ModelMultimodal Large Language Model+4Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention
In this paper, we present a novel approach for the task of eXplainable Question Answering (XQA), i.e., generating natural language (NL) explanations for the Visual Question Answering (VQA) problem. We generate NL explana…
Explanation GenerationLanguage ModelingLanguage ModellingQuestion Answering+2PolySmart @ TRECVid 2024 Medical Video Question Answering
Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical qu…
Question AnsweringRetrievalText RetrievalVideo Question Answering+1