ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model
Fine-grained multimodal capability in Multimodal Large Language Models (MLLMs) has emerged as a critical research direction, particularly for tackling the visual grounding (VG) problem. Despite the strong performance achieved by existing approaches, they often employ disparate design choices when fine-tuning MLLMs for VG, lacking systematic verification to support these designs. To bridge this gap, this paper presents a comprehensive study of various design choices that impact the VG performance of MLLMs. We conduct our analysis using LLaVA-1.5, which has been widely adopted in prior empirical studies of MLLMs. While more recent models exist, we follow this convention to ensure our findings remain broadly applicable and extendable to other architectures. We cover two key aspects: (1) exploring different visual grounding paradigms in MLLMs, identifying the most effective design, and providing our insights; and (2) conducting ablation studies on the design of grounding data to optimize MLLMs' fine-tuning for the VG task. Finally, our findings contribute to a stronger MLLM for VG, achieving improvements of +5.6% / +6.9% / +7.0% on RefCOCO/+/g over the LLaVA-1.5.
Code (0)
등록된 구현이 없습니다.
Tasks
Visual GroundingSimilar Papers 제목 키워드 기반
Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
Visual Grounding benchmarks assume that the object described by a referring expression is always present in the image, and grounding models are therefore rarely evaluated under semantically mismatched captions. In such c…
Referring ExpressionVisual GroundingInvestigating Compositional Challenges in Vision-Language Models for Visual Grounding
Pre-trained vision-language models (VLMs) have achieved high performance on various downstream tasks which have been widely used for visual grounding tasks in a weakly supervised manner. However despite the performan…
AttributeRelationVisual GroundingAttending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers
The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Tr…
Language ModelingLanguage ModellingVisual GroundingmRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
Large Vision-Language Models (LVLMs) have made remarkable strides in multimodal tasks such as visual question answering, visual grounding, and complex reasoning. However, they remain limited by static training data, susc…
Question AnsweringRAGRe-RankingRetrieval+3From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
3D visual grounding has made notable progress in localizing objects within complex 3D scenes. However, grounding referring expressions beyond objects in 3D scenes remains unexplored. In this paper, we introduce Anywhere3…
3D visual groundingObjectReferring ExpressionSpatial Reasoning+1