Adversarial Robustness for Visual Grounding of Multimodal Large Language Models

2024-05-16 · Kuofeng Gao, Yang Bai, Jiawang Bai, Yong Yang, Shu-Tao Xia

Multi-modal Large Language Models (MLLMs) have recently achieved enhanced performance across various vision-language tasks including visual grounding capabilities. However, the adversarial robustness of visual grounding remains unexplored in MLLMs. To fill this gap, we use referring expression comprehension (REC) as an example task in visual grounding and propose three adversarial attack paradigms as follows. Firstly, untargeted adversarial attacks induce MLLMs to generate incorrect bounding boxes for each object. Besides, exclusive targeted adversarial attacks cause all generated outputs to the same target bounding box. In addition, permuted targeted adversarial attacks aim to permute all bounding boxes among different objects within a single image. Extensive experiments demonstrate that the proposed methods can successfully attack visual grounding capabilities of MLLMs. Our methods not only provide a new perspective for designing novel attacks but also serve as a strong baseline for improving the adversarial robustness for visual grounding of MLLMs.

📄 PDF Abstract BibTeX arXiv:2405.09981

Code (1)

KuofengGao/MLLM-Grounding-Robustness 공식 구현 pytorch

Tasks

Adversarial Attack Adversarial Robustness Referring ExpressionReferring Expression ComprehensionVisual Grounding

Does language help generalization in vision models?

2021-04-16 · CoNLL (EMNLP) 2021 11 · Benjamin Devillers, Bhavin Choksi, Romain Bielawski, Rufin VanRullen

Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This co…

Adversarial Robustness Clustering Few-Shot Learning Transfer Learning+1

Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding

2025-11-21 · Daniil Ignatev, Ayman Santeer, Albert Gatt, Denis Paperno arxiv

We propose a zero-shot method for Natural Language Inference (NLI) that leverages multimodal representations by grounding language in visual contexts. Our approach generates visual representations of premises using text-…

Natural Language Understanding Natural Language Inference Visual Question Answering Visual Grounding

Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models

2026-02-10 · Xinwei Zhang, Li Bai, Tianwei Zhang, Youqian Zhang 외 arxiv

Large vision-language models (LVLMs) have achieved impressive performance across multimodal tasks, but their reliance on visual inputs exposes them to adversarial threats. Encoder-based attacks provide an efficient alter…

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

2025-07-09 · Chengfei Wu, Ronald Seoh, Bingxuan Li, Liqiang Zhang 외

Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual…

Diagnostic Multimodal Reasoning Question Answering Visual Question Answering+1

Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

2026-06-02 · Hashmat Shadab Malik, Muzammal Naseer, Salman Khan arxiv

Multimodal Large Language Models integrate visual perception into language reasoning, introducing a continuous attack surface susceptible to adversarial attacks. Prior work on MLLM robustness has focused largely on Engli…

Adversarial Robustness