Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering
Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a two-stage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved content. However, the retrieved knowledge is often inadequate. Retrievals are frequently too general and fail to cover specific knowledge needed to answer the question. Also, the naturally available supervision (whether the passage contains the correct answer) is weak and does not guarantee question relevancy. To address these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that provides stronger supervision during training and recognizes question-relevant entities to help retrieve more specific knowledge. Experiments show that our EnFoRe model achieves superior retrieval performance on OK-VQA, the currently largest outside-knowledge VQA dataset. We also combine the retrieved knowledge with state-of-the-art VQA models, and achieve a new state-of-the-art performance on OK-VQA.
Code (0)
등록된 구현이 없습니다.
Tasks
Passage RetrievalQuestion AnsweringRetrievalVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Passage Retrieval for Outside-Knowledge Visual Question Answering
In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowled…
Image CaptioningObjectPassage RetrievalQuestion Answering+3Phrase Retrieval Learns Passage Retrieval, Too
Dense retrieval methods have shown great promise over sparse retrieval methods in a range of NLP problems. Among them, dense phrase retrieval-the most fine-grained retrieval unit-is appealing because phrases can be direc…
Entity LinkingOpen-Domain Question AnsweringPassage RetrievalQuantization+4Topic-DPR: Topic-based Prompts for Dense Passage Retrieval
Prompt-based learning's efficacy across numerous natural language processing tasks has led to its integration into dense passage retrieval. Prior research has mainly focused on enhancing the semantic understanding of pre…
Contrastive LearningPassage RetrievalRetrievalMIA 2022 Shared Task Submission: Leveraging Entity Representations, Dense-Sparse Hybrids, and Fusion-in-Decoder for Cross-Lingual Question Answering
We describe our two-stage system for the Multilingual Information Access (MIA) 2022 Shared Task on Cross-Lingual Open-Retrieval Question Answering. The first stage consists of multilingual passage retrieval with a hybrid…
Cross-Lingual Question AnsweringDecoderLanguage ModelingLanguage Modelling+3Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering
This paper studies a category of visual question answering tasks, in which accessing external knowledge is necessary for answering the questions. This category is called outside-knowledge visual question answering (OK-VQ…
Passage RetrievalQuestion AnsweringRetrievalVisual Question Answering+1