Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base
Knowledge-based visual question answering (KVQA) task aims to answer questions that require additional external knowledge as well as an understanding of images and questions. Recent studies on KVQA inject an external knowledge in a multi-modal form, and as more knowledge is used, irrelevant information may be added and can confuse the question answering. In order to properly use the knowledge, this study proposes the following: 1) We introduce a novel semantic inconsistency measure using caption uncertainty and semantic similarity. 2) We suggest a new external knowledge assimilation method based on the semantic inconsistency measure and apply it to integrate explicit knowledge and implicit knowledge for KVQA. 3) The proposed method is evaluated on the OK-VQA dataset and achieves the state-of-the-art performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSemantic SimilaritySemantic Textual SimilarityVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base
Knowledge-based visual question answering (KVQA) task aims to answer questions that require additional external knowledge as well as an understanding of images and questions. Recent studies on KVQA inject an external kno…
Question AnsweringSemantic SimilaritySemantic Textual SimilarityVisual Question Answering+1V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering
Multimodal Large Language Models (MLLMs) have shown remarkable capability in assisting disease diagnosis in medical visual question answering (VQA). However, their outputs remain vulnerable to hallucinations (i.e., respo…
Visual Question AnsweringComputational EfficiencyUncertainty-aware Language Modeling for Selective Question Answering
We present an automatic large language model (LLM) conversion approach that produces uncertainty-aware LLMs capable of estimating uncertainty with every prediction. Our approach is model- and data-agnostic, is computatio…
Language ModelingLanguage ModellingLarge Language ModelQuestion Answering+1Estimating semantic structure for the VQA answer space
Since its appearance, Visual Question Answering (VQA, i.e. answering a question posed over an image), has always been treated as a classification problem over a set of predefined answers. Despite its convenience, this cl…
General ClassificationQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Uncertainty Quantification in Retrieval Augmented Question Answering
Retrieval augmented Question Answering (QA) helps QA models overcome knowledge gaps by incorporating retrieved evidence, typically a set of passages, alongside the question at test time. Previous studies show that this a…
Question AnsweringRetrievalUncertainty Quantification