3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V
In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Object Detectionobject-detectionObject DetectionVisual PromptingSimilar Papers 제목 키워드 기반
GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
Visual generation models have made remarkable progress in creating realistic images from text prompts, yet struggle with complex prompts that specify multiple objects with precise spatial relationships and attributes. Ef…
AttributeImage Generationreinforcement-learningReinforcement Learning+1Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability
Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. Previous paradigms either explore better scoring functions or utilize the kno…
Out-of-Distribution DetectionNext-Generation Conflict Forecasting: Unleashing Predictive Patterns through Spatiotemporal Learning
Forecasting violent conflict at high spatial and temporal resolution remains a central challenge for both researchers and policymakers. This study presents a novel neural network architecture for forecasting three distin…
Feature EngineeringHumanitarianUnleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem
We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess immense reasoning potential inherited from the pre-training stage. With reinforcement learning (RL), these models can improve dramatically on rea…
GPUMathReinforcement Learning (RL)Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions ty…
Scene UnderstandingSpatial ReasoningVideo Generation