paper-with-me

홈 › Papers

Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints

2026-03-08 · Chenxi Li, Xianggan Liu, Dake Shen, Yaosong Du, Zhibo Yao, Hao Jiang, Linyi Jiang, Chengwei Cao, Jingzhe Zhang, RanYi Peng, Peiling Bai, Xiande Huang arxiv

Despite the rapid progress of Large Vision-Language Models (LVLMs), the integration of visual modalities introduces new safety vulnerabilities that adversaries can exploit to elicit biased or malicious outputs. In this paper, we demonstrate an underexplored vulnerability via semantic slot filling, where LVLMs complete missing slot values with unsafe content even when the slot types are deliberately crafted to appear benign. Building on this finding, we propose StructAttack, a simple yet effective single-query jailbreak framework under black-box settings. StructAttack decomposes a harmful query into a central topic and a set of benign-looking slot types, then embeds them as structured visual prompts (e.g., mind maps, tables, or sunburst diagrams) with small random perturbations. Paired with a completion-guided instruction, LVLMs automatically recompose the concealed semantics and generate unsafe outputs without triggering safety mechanisms. Although each slot appears benign in isolation (local benignness), StructAttack exploits LVLMs' reasoning to assemble these slots into coherent harmful semantics. Extensive experiments on multiple models and benchmarks show the efficacy of our proposed StructAttack.

📄 PDF Abstract BibTeX arXiv:2603.07590

Code (0)

등록된 구현이 없습니다.

Tasks

Slot Filling

Similar Papers 제목 키워드 기반

Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks

2022-03-25 · Jiacong Hu, Jing Gao, Jingwen Ye, Yang Gao 외

With the rapid development of deep learning, the increasing complexity and scale of parameters make training a new model increasingly resource-intensive. In this paper, we start from the classic convolutional neural netw…

Incremental LearningKnowledge DistillationModel CompressionTransfer Learning

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

2024-09-24 · Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li 외

Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modu…

Structurally Different Neural Network Blocks for the Segmentation of Atrial and Aortic Perivascular Adipose Tissue in Multi-centre CT Angiography Scans

2023-06-06 · Ikboljon Sobirov, Cheng Xie, Muhammad Siddique, Parijat Patel 외

Since the emergence of convolutional neural networks (CNNs) and, later, vision transformers (ViTs), deep learning architectures have predominantly relied on identical block types with varying hyperparameters. We propose …

DiagnosticImage SegmentationMedical Image Segmentationmodel+3

LEGO-ABSA: A Prompt-based Task Assemblable Unified Generative Framework for Multi-task Aspect-based Sentiment Analysis

2022-10-01 · COLING 2022 10 · Tianhao Gao, Jun Fang, Hanyu Liu, Zhiyuan Liu 외

Aspect-based sentiment analysis (ABSA) has received increasing attention recently. ABSA can be divided into multiple tasks according to the different extracted elements. Existing generative methods usually treat the outp…

Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Sentiment Analysis

Lego Sketch: A Scalable Memory-augmented Neural Network for Sketching Data Streams

2025-05-26 · Yuan Feng, Yukun Cao, Hairu Wang, Xike Xie 외

Sketches, probabilistic structures for estimating item frequencies in infinite data streams with limited space, are widely used across various domains. Recent studies have shifted the focus from handcrafted sketches to n…