Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

2025-10-26 · Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan arxiv

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.

📄 PDF Abstract BibTeX arXiv:2510.22694

Code (0)

등록된 구현이 없습니다.

MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering

2026-02-01 · Hui Wu, Haoquan Zhai, Yuchen Li, Hengyi Cai 외 arxiv

Retrieval-based multimodal document QA aims to identify and integrate relevant information from visually rich documents with complex multimodal structures. While retrieval-augmented generation (RAG) has shown strong perf…

Question Answering

RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training

2024-10-18 · Muhe Ding, Yang Ma, Pengda Qin, Jianlong Wu 외

Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. MLLMs involve significant external k…

Denoising Question Answering Retrieval

Self-adaptive Multimodal Retrieval-Augmented Generation

2024-10-15 · Wenjia Zhai

Traditional Retrieval-Augmented Generation (RAG) methods are limited by their reliance on a fixed number of retrieved documents, often resulting in incomplete or noisy information that undermines task performance. Althou…

Image Captioning RAG Response Generation Retrieval+1

Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

2024-11-05 · Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang 외

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typicall…

Benchmarking Hallucination Question Answering Retrieval+3

Unveiling the Potential of Multimodal Retrieval Augmented Generation with Planning

2025-01-26 · Xiaohan Yu, Zhihan Yang, Chong Chen

Multimodal Retrieval Augmented Generation (MRAG) systems, while promising for enhancing Multimodal Large Language Models (MLLMs), often rely on rigid, single-step retrieval methods. This limitation hinders their ability …

Retrieval Retrieval-augmented Generation