ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding
3D visual grounding (3DVG) involves localizing entities in a 3D scene referred to by natural language text. Such models are useful for embodied AI and scene retrieval applications, which involve searching for objects or patterns using natural language descriptions. While recent works have focused on LLM-based scaling of 3DVG datasets, these datasets do not capture the full range of potential prompts which could be specified in the English language. To ensure that we are scaling up and testing against a useful and representative set of prompts, we propose a framework for linguistically analyzing 3DVG prompts and introduce Visual Grounding with Diverse Language in 3D (ViGiL3D), a diagnostic dataset for evaluating visual grounding methods against a diverse set of language patterns. We evaluate existing open-vocabulary 3DVG methods to demonstrate that these methods are not yet proficient in understanding and identifying the targets of more challenging, out-of-distribution prompts, toward real-world applications.
Code (0)
등록된 구현이 없습니다.
Tasks
3D visual groundingDiagnosticVisual GroundingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Scaling Diverse Language Generation for 3D Visual Grounding
Developing robust models for 3D visual grounding (3DVG), the localization of entities in a 3D scene described in natural language, is important for enabling agents to correspond spatial language with objects in the physi…
Visual GroundingStaying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain …
Visual GroundingThink Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation
Referring Audio-Visual Segmentation (Ref-AVS) aims to segment target objects in audible videos based on given reference expressions. Prior works typically rely on learning latent embeddings via multimodal fusion to promp…
SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation
This paper introduces SignAgent, a novel agentic framework that utilises Large Language Models (LLMs) for scalable, linguistically-grounded Sign Language (SL) annotation and dataset curation. Traditional computational me…
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes
Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness freque…
Multimodal ReasoningVisual ReasoningVisual Grounding