Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR
Visual Commonsense Reasoning (VCR) calls for explanatory reasoning behind question answering over visual scenes. To achieve this goal, a model is required to provide an acceptable rationale as the reason for the predicted answers. Progress on the benchmark dataset stems largely from the recent advancement of Vision-Language Transformers (VL Transformers). These models are first pre-trained on some generic large-scale vision-text datasets, and then the learned representations are transferred to the downstream VCR task. Despite their attractive performance, this paper posits that the VL Transformers do not exhibit visual commonsense, which is the key to VCR. In particular, our empirical results pinpoint several shortcomings of existing VL Transformers: small gains from pre-training, unexpected language bias, limited model architecture for the two inseparable sub-tasks, and neglect of the important object-tag correlation. With these findings, we tentatively suggest some future directions from the aspect of dataset, evaluation metric, and training tricks. We believe this work could make researchers revisit the intuition and goals of VCR, and thus help tackle the remaining challenges in visual reasoning.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringTAGVisual Commonsense ReasoningVisual ReasoningSimilar Papers 제목 키워드 기반
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part…
Image GenerationNatural Language UnderstandingPositionThings not Written in Text: Exploring Spatial Commonsense from Visual Signals
Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part…
Image GenerationNatural Language UnderstandingPositionVL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers
Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems. However, although visualization and interpretability tools have become available for NLP model…
Question AnsweringVisual Commonsense ReasoningVisual Question AnsweringVisual Question Answering (VQA)ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models
In our work, we explore the synergistic capabilities of pre-trained vision-and-language models (VLMs) and large language models (LLMs) on visual commonsense reasoning (VCR) problems. We find that VLMs and LLMs-based deci…
Image CaptioningVisual Commonsense ReasoningVCD: Knowledge Base Guided Visual Commonsense Discovery in Images
Visual commonsense contains knowledge about object properties, relationships, and behaviors in visual data. Discovering visual commonsense can provide a more comprehensive and richer understanding of images, and enhance …
Decision MakingLanguage ModellingQuestion AnsweringVisual Question Answering