Can Language Understand Depth?

2022-07-03 · Renrui Zhang, Ziyao Zeng, Ziyu Guo, Yafeng Li

Besides image classification, Contrastive Language-Image Pre-training (CLIP) has accomplished extraordinary success for a wide range of vision tasks, including object-level and 3D space understanding. However, it's still challenging to transfer semantic knowledge learned from CLIP into more intricate tasks of quantified targets, such as depth estimation with geometric information. In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of the input image could respond to a certain semantic distance token and then be projected to a quantified depth bin for coarse estimation. Without any training, our DepthCLIP surpasses existing unsupervised methods and even approaches the early fully-supervised networks. To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation. We hope our work could cast a light on future research. The code is available at https://github.com/Adonis-galaxy/DepthCLIP.

📄 PDF Abstract BibTeX arXiv:2207.01077

Code (1)

adonis-galaxy/depthclip 공식 구현 pytorch

Tasks

Depth Estimation image-classification Image Classification Monocular Depth Estimation

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

SpatialBot: Precise Spatial Understanding with Vision Language Models

2024-06-19 · Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li 외

Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose…

Spatial Reasoning

DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model

2026-03-06 · Hao Yang, Hongbo Zhang, Yanyan Zhao, Bing Qin arxiv

Multimodal large language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth info…

Scene Understanding Spatial Reasoning Image Captioning

Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

2026-07-01 · Yiqian Liu, Iuliia Kotseruba, John K. Tsotsos arxiv

In this paper, we study depth perception of vision-language models (VLMs) to isolate the effects of pictorial depth cues and disentangle vision and language influences on model performance. To this end, we combine depth-…

Referring Expression Image Generation

DepthLM: Metric Depth From Vision Language Models

2025-09-29 · Zhipeng Cai, Ching-Feng Yeh, Hu Xu, Zhuang Liu 외 arxiv

Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D f…

Depth Estimation

Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis

2024-01-31 · Jianing Li, Xi Nan, Ming Lu, Li Du 외

Multi-modal large language models (MLLMs) have demonstrated remarkable vision-language capabilities, primarily due to the exceptional in-context understanding and multi-task learning strengths of large language models (L…

Multi-Task Learning Question Answering Visual Question Answering (VQA)