Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering
Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge. In this paper, we call for an alternative paradigm for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the natural language space. This paradigm takes advantage of the sheer volume of gigantic knowledge bases and the richness of pre-trained language models. A Transform-Retrieve-Generate framework (TRiG) framework is proposed, which can be plug-and-played with alternative image-to-text models and textual knowledge bases. Experimental results show that our TRiG framework outperforms all state-of-the-art supervised methods by at least 11.1% absolute margin.
Code (0)
등록된 구현이 없습니다.
Tasks
Generative Question AnsweringImage to textPassage RetrievalQuestion AnsweringRetrievalVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering
Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous wo…
Generative Question AnsweringImage to textPassage RetrievalQuestion Answering+3Training-Induced Bias Toward LLM-Generated Content in Dense Retrieval
Dense retrieval is a promising approach for acquiring relevant context or world knowledge in open-domain natural language processing tasks and is now widely used in information retrieval applications. However, recent rep…
Information RetrievalNatural QuestionsA New Paradigm of User-Centric Wireless Communication Driven by Large Language Models
The next generation of wireless communications seeks to deeply integrate artificial intelligence (AI) with user-centric communication networks, with the goal of developing AI-native networks that more accurately address …
Semantic CommunicationEncode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception
We depend on our own memory to encode, store, and retrieve our experiences. However, memory lapses can occur. One promising avenue for achieving memory augmentation is through the use of augmented reality head-mounted di…
Can a Frozen Pretrained Language Model be used for Zero-shot Neural Retrieval on Entity-centric Questions?
Neural document retrievers, including dense passage retrieval (DPR), have outperformed classical lexical-matching retrievers, such as BM25, when fine-tuned and tested on specific question-answering datasets. However, it …
Domain GeneralizationLanguage ModelingLanguage ModellingPassage Retrieval+2