Do We Really Need a Large Number of Visual Prompts?

2023-05-26 · Youngeun Kim, Yuhang Li, Abhishek Moitra, Ruokai Yin, Priyadarshini Panda

Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.

📄 PDF Abstract BibTeX arXiv:2305.17223

Code (0)

등록된 구현이 없습니다.

Tasks

Transfer Learning Visual Prompt Tuning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Residual Connection 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

Scaling Down Text Encoders of Text-to-Image Diffusion Models

2025-03-25 · CVPR 2025 1 · Lifu Wang, Daqing Liu, Xinchen Liu, Xiaodong He

Text encoders in diffusion models have rapidly evolved, transitioning from CLIP to T5-XXL. Although this evolution has significantly enhanced the models' ability to understand complex prompts and generate text, it also l…

GPU Image Generation Knowledge Distillation Text to Image Generation+1

Do Prompt-Based Models Really Understand the Meaning of Their Prompts?

2022-01-16 · ACL ARR January 2022 1 · Anonymous

Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. Such success can give the impression that prompts help models to learn faster in the same w…

Few-Shot Learning Natural Language Inference

Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

2016-03-17 · Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan 외

Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or sh…

Is the System Message Really Important to Jailbreaks in Large Language Models?

2024-02-20 · Xiaotian Zou, Yongkang Chen, Ke Li

The rapid evolution of Large Language Models (LLMs) has rendered them indispensable in modern society. While security measures are typically to align LLMs with human values prior to release, recent studies have unveiled …

Evolutionary Algorithms

LLM-based Affective Text Generation Quality Based on Different Quantization Values

2025-01-31 · Yarik Menchaca Resendiz, Roman Klinger

Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a lar…

GPU Quantization Text Generation