Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

2026-08-21 · Rujin Liang, Zhongpu Chen, Yuhao Lei, Xin Miao arxiv

While multimodal retrieval-augmented generation (RAG) systems increasingly rely on images as external knowledge sources, the introduction of poisoned visual evidence can severely compromise multimodal large language model (MLLM) generation. Unlike prior attacks that rely on altering textual metadata, we introduce Vis-Poison, a novel visual knowledge poisoning attack where the poisoned image itself is the attacker-controlled payload, without manipulating captions, summaries, metadata, or other associated text. Specifically, this attack is instantiated through an automated multi-agent method that constructs visually plausible poisoned images. To assess its impact, we evaluate Vis-Poison across two representative multimodal RAG pipelines, four embedding models, and six generation models. Empirically, Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases in \emph{black-box} settings. Moreover, Vis-Poison remains effective against various MLLMs that can answer correctly from parametric knowledge alone, with an average success rate above 60%. Code and data are available at https://github.com/SWUFE-DB-Group/Vis-Poison.

📄 PDF Abstract BibTeX arXiv:2608.20756

Code (0)

등록된 구현이 없습니다.

Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation

2026-02-26 · Kennedy Edemacu, Mohammad Mahdi Shokri arxiv

Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for enhancing multimodal large language models by grounding their responses in external, factual knowledge and thus mitigating hallucinations. Howev…

Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation

2025-03-08 · Yinuo Liu, Zenghui Yuan, Guiyao Tie, Jiawen Shi 외

Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning capability of vision-language models (VLMs) by dynamically accessing information from external knowledge bases. In this work, we introduce \te…

RAG Retrieval Retrieval-augmented Generation Visual Reasoning

MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks

2025-02-25 · Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios 외

Multimodal large language models (MLLMs) equipped with Retrieval Augmented Generation (RAG) leverage both their rich parametric knowledge and the dynamic, external knowledge to excel in tasks such as Question Answering. …

Misinformation Question Answering RAG Retrieval-augmented Generation

Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation

2026-05-11 · Peiru Yang, Haoran Zheng, Tong Ju, Shiting Wang 외 arxiv

Retrieval-augmented generation (RAG) is a widely adopted paradigm for enhancing LLMs in medical applications by incorporating expert multimodal knowledge during generation. However, the underlying retrieval databases may…

Medical Diagnosis

Data Poisoning Attacks Against Multimodal Encoders

2022-09-30 · Ziqing Yang, Xinlei He, Zheng Li, Michael Backes 외

Recently, the newly emerged multimodal models, which leverage both visual and linguistic modalities to train powerful encoders, have gained increasing attention. However, learning from a large-scale unlabeled dataset als…

Contrastive Learning Data Poisoning