Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering

2026-01-14 · Jongha Kim, Byungoh Ko, Jeehye Na, Jinsung Yoon, Hyunwoo J. Kim arxiv

Despite the remarkable capabilities of Large Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities. Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the retrieval results, consistently performing the best across the weakest to the strongest retrieval results. Code is available at https://github.com/mlvlab/RMCD.

📄 PDF Abstract BibTeX arXiv:2602.06050

Code (0)

등록된 구현이 없습니다.

Tasks

Visual Question Answering

Steering Multimodal Large Language Models Decoding for Context-Aware Safety

2025-09-23 · Zheyuan Liu, Zhangchen Xu, Guangyao Dou, Xiangchi Yuan 외 arxiv

Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitiv…

From Context-Aware to Conflict-Aware: Generalizing Contrastive Decoding for Knowledge Conflict in LLMs

2026-06-09 · Runze Jiang, Taiqiang Wu, Yan Wang, Bingyu Zhu 외 arxiv

When large language models generate from retrieved or augmented contexts, conflicts between external context and parametric priors remain a central reliability bottleneck. Existing contrastive decoding methods follow a \…

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

2026-01-29 · Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro arxiv

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental defic…

Multimodal Reasoning

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

2026-06-29 · Yuqing Lei, Wenbo Lyu, Yingjun Du, Xiantong Zhen 외 arxiv

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions …

Visual Grounding

Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generation

2026-07-01 · Raymond Li, Md Tawkat Islam Khondaker, Amirhossein Abaskohi, Gabriel Murray 외 arxiv

Retrieval-augmented generation (RAG) increasingly requires models to answer questions from multiple retrieved documents, where only some sources are relevant and the retrieved bundle may contain stale, noisy, or conflict…

Question Answering