Large Language Models Can Understanding Depth from Monocular Images
Monocular depth estimation is a critical function in computer vision applications. This paper shows that large language models (LLMs) can effectively interpret depth with minimal supervision, using efficient resource utilization and a consistent neural network architecture. We introduce LLM-MDE, a multimodal framework that deciphers depth through language comprehension. Specifically, LLM-MDE employs two main strategies to enhance the pretrained LLM's capability for depth estimation: cross-modal reprogramming and an adaptive prompt estimation module. These strategies align vision representations with text prototypes and automatically generate prompts based on monocular images, respectively. Comprehensive experiments on real-world MDE datasets confirm the effectiveness and superiority of LLM-MDE, which excels in few-/zero-shot tasks while minimizing resource use. The source code is available.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationMonocular Depth EstimationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Center3D: Center-based Monocular 3D Object Detection with Joint Depth Understanding
Localizing objects in 3D space and understanding their associated 3D properties is challenging given only monocular RGB images. The situation is compounded by the loss of depth information during perspective projection. …
3D Object DetectionDepth EstimationGeneral ClassificationMonocular 3D Object Detection+3SynDeMo: Synergistic Deep Feature Alignment for Joint Learning of Depth and Ego-Motion
Despite well-established baselines, learning of scene depth and ego-motion from monocular video remains an ongoing challenge, specifically when handling scaling ambiguity issues and depth inconsistencies in image sequenc…
Depth EstimationMotion EstimationMGNet: Monocular Geometric Scene Understanding for Autonomous Driving
We introduce MGNet, a multi-task framework for monocular geometric scene understanding. We define monocular geometric scene understanding as the combination of two known tasks: Panoptic segmentation and self-supervised m…
Autonomous DrivingDepth EstimationGPUMonocular Depth Estimation+2Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learnin…
Depth EstimationGPUMonocular Depth EstimationScene Understanding+1DGOcc: Depth-aware Global Query-based Network for Monocular 3D Occupancy Prediction
Monocular 3D occupancy prediction, aiming to predict the occupancy and semantics within interesting regions of 3D scenes from only 2D images, has garnered increasing attention recently for its vital role in 3D scene unde…
GPUPredictionScene Understanding