Token Coordinated Prompt Attention is Needed for Visual Prompting

2025-05-05 · Zichen Liu, Xu Zou, Gang Hua, Jiahuan Zhou

Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existing methods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance. To address this issue, we propose a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts to different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens-global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Furthermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features. Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of the extracted features. The code is available at https://github.com/zhoujiahuan1991/ICML2025-TCPA.

📄 PDF Abstract BibTeX arXiv:2505.02406

Code (1)

zhoujiahuan1991/icml2025-tcpa 공식 구현 pytorch

Tasks

DiversityVisual Prompting

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

2026-06-18 · Yang Tan, Junlong Tong, Linan Yue, Hao Wu 외 arxiv

Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods l…

How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

2026-05-09 · YiJie Huang, Yiqun Zhang, Zhuoyue Jia, Xiaocui Yang 외 arxiv

Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cost raises a basic scaling question: as multimodal models grow, how many…

[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs

2024-12-08 · Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin 외

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across a wide range of vision-language tasks, garnering significant attention in the computer vision. However, their efficient deploy…

ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models

2024-07-31 · Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li 외

In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test-time optimization of a learnable latent variable. We observe that attention, as the core…

Domain Generalization Prompt Learning

KNN Transformer with Pyramid Prompts for Few-Shot Learning

2024-10-14 · Wenhao Li, Qiangchang Wang, Peng Zhao, Yilong Yin

Few-Shot Learning (FSL) aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features. However, they usua…

Few-Shot Learning