I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction
Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.
Code (0)
등록된 구현이 없습니다.
Tasks
3D ReconstructionSpatial ReasoningVisual ReasoningSimilar Papers 제목 키워드 기반
Enhancing Spatial Reasoning through Visual and Textual Thinking
The spatial reasoning task aims to reason about the spatial relationships in 2D and 3D space, which is a fundamental capability for Visual Question Answering (VQA) and robotics. Although vision language models (VLMs) hav…
Visual Question AnsweringSpatial ReasoningLogical ReasoningSeeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning
Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes. This paper addresses two key challenges: (1) conve…
Relation ExtractionScene UnderstandingSpatial Knowledge Distillation to aid Visual Reasoning
For tasks involving language and vision, the current state-of-the-art methods tend not to leverage any additional information that might be present to gather relevant (commonsense) knowledge. A representative task is Vis…
DiagnosticKnowledge DistillationQuestion AnsweringRelational Reasoning+3MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation
When embodied AI is expanding from traditional object detection and recognition to more advanced tasks of robot manipulation and actuation planning, visual spatial reasoning from the video inputs is necessary to perceive…
Robot ManipulationSpatial ReasoningObject DetectionSpatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning
Vision-language models (VLMs) have advanced multimodal reasoning but still face challenges in spatial reasoning for 3D scenes and complex object configurations. To address this, we introduce SpatialViLT, an enhanced VLM …
Multimodal ReasoningMulti-Task LearningSpatial Reasoning