MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
Recently, Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often struggle to ground fine-grained visual concepts in complex scenes. In this paper, we propose MoDA (Modulation Adapter), a lightweight yet effective module designed to refine pre-aligned visual features through instruction-guided modulation. Our approach follows the standard LLaVA training protocol, consisting of a two-stage process: (1) aligning image features to the LLMs input space via a frozen vision encoder and adapter layers, and (2) refining those features using the MoDA adapter during the instructional tuning stage. MoDA employs a Transformer-based cross-attention mechanism to generate a modulation mask over the aligned visual tokens, thereby emphasizing semantically relevant embedding dimensions based on the language instruction. The modulated features are then passed to the LLM for autoregressive language generation. Our experimental evaluation shows that MoDA improves visual grounding and generates more contextually appropriate responses, demonstrating its effectiveness as a general-purpose enhancement for image-based MLLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Instruction FollowingText GenerationVisual GroundingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning
Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate…
Few-Shot LearningTransfer LearningHiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual or linguistic k…
cross-modal alignmentVisual GroundingFine-Grained Scene Image Classification with Modality-Agnostic Adapter
When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately design…
image-classificationImage ClassificationTaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life
Accurately generating images across the Tree of Life is difficult: there are over 10M distinct species on Earth, many of which differ only by subtle visual traits. Despite the remarkable progress in text-to-image synthes…
Image GenerationFine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
Adapter-based approaches have garnered attention for fine-tuning pre-trained Vision-Language Models (VLMs) on few-shot classification tasks. These methods strive to develop a lightweight module that better aligns visual …
Few-Shot Learning