Exploring Spatial Language Grounding Through Referring Expressions
Spatial Reasoning is an important component of human cognition and is an area in which the latest Vision-language models (VLMs) show signs of difficulty. The current analysis works use image captioning tasks and visual question answering. In this work, we propose using the Referring Expression Comprehension task instead as a platform for the evaluation of spatial reasoning by VLMs. This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not'). In our analysis, we use task-specific architectures as well as large VLMs and highlight their strengths and weaknesses in dealing with these specific situations. While all these models face challenges with the task at hand, the relative behaviors depend on the underlying models and the specific categories of spatial semantics (topological, directional, proximal, etc.). Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningNegationobject-detectionObject DetectionQuestion AnsweringReferring ExpressionReferring Expression ComprehensionSentenceSpatial ReasoningVisual Question AnsweringSimilar Papers 제목 키워드 기반
STORM: End-to-End Referring Multi-Object Tracking in Videos
Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding …
Multi-Object TrackingGrounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
The human language is one of the most natural interfaces for humans to interact with robots. This paper presents a robot system that retrieves everyday objects with unconstrained natural language descriptions. A core iss…
ObjectProgressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
Referring image segmentation aims to localize and segment a target object in an image based on a free-form referring expression. The core challenge lies in effectively bridging linguistic descriptions with object-level v…
Semantic SegmentationInstance SegmentationReferring ExpressionImage SegmentationChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision and language models, yet most prior work focuses on natural images. In …
Instance SegmentationReferring ExpressionVisual GroundingViewpoint-Aware Visual Grounding in 3D Scenes
Referring expressions for visual objects often include descriptions of relative spatial arrangements to other objects -- e.g. "to the right of" -- that depend on the point of view of the speaker. In 2D referring expr…
3D visual groundingReferring ExpressionVisual Grounding