WinoViz: Probing Visual Properties of Objects Under Different States
Humans perceive and comprehend different visual properties of an object based on specific contexts. For instance, we know that a banana turns brown `when it becomes rotten,'' whereas it appears green `when it is unripe.'' Previous studies on probing visual commonsense knowledge have primarily focused on examining language models' understanding of typical properties (e.g., colors and shapes) of objects. We present WinoViz, a text-only evaluation dataset, consisting of 1,380 examples that probe the reasoning abilities of language models regarding variant visual properties of objects under different contexts or states. Our task is challenging since it requires pragmatic reasoning (finding intended meanings) and visual knowledge reasoning. We also present multi-hop data, a more challenging version of our data, which requires multi-step reasoning chains to solve our task. In our experimental analysis, our findings are: a) Large language models such as GPT-4 demonstrate effective performance, but when it comes to multi-hop data, their performance is significantly degraded. b) Large models perform well on pragmatic reasoning, but visual knowledge reasoning is a bottleneck in our task. c) Vision-language models outperform their language-model counterparts. d) A model with machine-generated images performs poorly in our task. This is due to the poor quality of the generated images.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case
Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the powe…
Probing Emergent Semantics in Predictive Agents via Question Answering
Recent work has shown how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to dec…
DecoderQuestion AnsweringUnderstanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions
In this paper, we consider the problem of understanding the physical properties of unseen objects through interactions between the objects and a robot. Handling unseen objects with special properties such as deformabilit…
Motion PlanningTask and Motion PlanningTask PlanningProbing the Role of Positional Information in Vision-Language Models
In most Vision-Language models (VL) the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we …
Contrastive LearningImage-text matchingObject LocalizationPosition+4Probing the Role of Positional Information in Vision-Language Models
In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we…
Contrastive LearningImage-text matchingObject LocalizationPosition+3