홈 › Papers

Learning to Ground VLMs without Forgetting

2024-10-14 · Aritra Bhowmik, Mohammad Mahdi Derakhshani, Dennis Koelma, Martin R. Oswald, Yuki M. Asano, Cees G. M. Snoek

Spatial awareness is key to enable embodied multimodal AI systems. Yet, without vast amounts of spatial supervision, current Visual Language Models (VLMs) struggle at this task. In this paper, we introduce LynX, a framework that equips pretrained VLMs with visual grounding ability without forgetting their existing image and language understanding skills. To this end, we propose a Dual Mixture of Experts module that modifies only the decoder layer of the language model, using one frozen Mixture of Experts (MoE) pre-trained on image and language understanding and another learnable MoE for new grounding capabilities. This allows the VLM to retain previously learned knowledge and skills, while acquiring what is missing. To train the model effectively, we generate a high-quality synthetic dataset we call SCouT, which mimics human reasoning in visual grounding. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process, thereby simplifying the task of visual grounding. We evaluate LynX on several object detection and visual grounding datasets, demonstrating strong performance in object detection, zero-shot localization and grounded reasoning while maintaining its original image and language understanding capabilities on seven standard benchmark datasets.

📄 PDF Abstract BibTeX arXiv:2410.10491

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Language ModellingMixture-of-ExpertsMultimodal Reasoning object-detection Object Detection Visual Grounding

Methods 이 논문이 사용한 방법론

MoE 설명 없음

Learning without Forgetting for Vision-Language Models

2023-05-30 · Da-Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye 외

Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world, which requires a learning system to adapt to new tasks without forgetting former ones. While traditional CIL methods focus…

class-incremental learning Class Incremental Learning Continual Learning Incremental Learning

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

2026-05-20 · Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng 외 arxiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent wi…

Sparse Spectral LoRA: Routed Experts for Medical VLMs

2026-04-01 · Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat, Yiming Xiao 외 arxiv

Large vision-language models (VLMs) excel on general benchmarks but often lack robustness in medical imaging, where heterogeneous supervision induces cross-dataset interference and sensitivity to data regime (i.e., how t…

Visual Question Answering

EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation

2026-03-10 · Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu 외 arxiv

The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge. While parameter-efficient prompt learning methods offer a promising path, they of…

Few-Shot Learning

APT: Atomic Physical Transitions for Causal Video-Language Understanding

2026-06-17 · Shang Wu, Haoran Lu, Songling Liu, Chenwei Xu 외 arxiv

Physical events are not understood by their names alone, but by the causal state changes that compose them. A clip-level label such as "bounce" can be correct while hiding the process that makes the event physically vali…