VQA-LOL: Visual Question Answering under the Lens of Logic
Logical connectives and their implications on the meaning of a natural language sentence are a fundamental aspect of understanding. In this paper, we investigate whether visual question answering (VQA) systems trained to answer a question about an image, are able to answer the logical composition of multiple such questions. When put under this \textit{Lens of Logic}, state-of-the-art VQA models have difficulty in correctly answering these logically composed questions. We construct an augmentation of the VQA dataset as a benchmark, with questions containing logical compositions and linguistic transformations (negation, disjunction, conjunction, and antonyms). We propose our {Lens of Logic (LOL)} model which uses question-attention and logic-attention to understand logical connectives in the question, and a novel Fr\'echet-Compatibility Loss, which ensures that the answers of the component questions and the composed question are consistent with the inferred logical operation. Our model shows substantial improvement in learning logical compositions while retaining performance on VQA. We suggest this work as a move towards robustness by embedding logical connectives in visual understanding.
Code (0)
등록된 구현이 없습니다.
Tasks
NegationQuestion AnsweringSentenceVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answering
Large language models are increasingly used to answer questions over annual reports, earnings decks, and analyst notes, yet their outputs remain difficult to verify in high-stakes financial workflows. A fluent answer can…
Question AnsweringMaXM: Towards Multilingual Visual Question Answering
Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper…
Question AnsweringTranslationVisual Question AnsweringVisual Question Answering (VQA)FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities …
Decision MakingLogical ReasoningQuestion AnsweringSpatial Reasoning+2Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of c…
Visual Question AnsweringVisual Question Answering based on Formal Logic
Visual question answering (VQA) has been gaining a lot of traction in the machine learning community in the recent years due to the challenges posed in understanding information coming from multiple modalities (i.e., ima…
Formal LogicQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)