Retrieval Augmented Visual Question Answering with Outside Knowledge

2022-10-07 · Weizhe Lin, Bill Byrne

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also introduce new diagnostic metrics to analyze how retrieval and generation interact. The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.

📄 PDF Abstract BibTeX arXiv:2210.03809

Code (1)

linweizhedragon/retrieval-augmented-visual-question-answering 공식 구현 pytorch

Tasks

Answer GenerationDiagnosticPassage Retrieval Question Answering Retrieval Visual Question Answering Visual Question Answering (VQA)

Passage Retrieval for Outside-Knowledge Visual Question Answering

2021-05-09 · Chen Qu, Hamed Zamani, Liu Yang, W. Bruce Croft 외

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowled…

Image Captioning Object Passage Retrieval Question Answering+3

Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

2022-10-18 · Jialin Wu, Raymond J. Mooney

Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a two-stage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved content…

Passage Retrieval Question Answering Retrieval Visual Question Answering+1

Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering

2023-06-28 · Alireza Salemi, Mahta Rafiee, Hamed Zamani

This paper studies a category of visual question answering tasks, in which accessing external knowledge is necessary for answering the questions. This category is called outside-knowledge visual question answering (OK-VQ…

Passage Retrieval Question Answering Retrieval Visual Question Answering+1

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

2026-07-02 · Valentin J. J. Kreileder, Johannes Reisinger, Andreas Fischer arxiv

Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves…

FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA

2025-02-25 · S M Sarwar

Visual Question Answering requires models to generate accurate answers by integrating visual and textual understanding. However, VQA models still struggle with hallucinations, producing convincing but incorrect answers, …

Question Answering Retrieval Retrieval-augmented Generation Visual Question Answering+1