TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
Vision Transformers (ViTs) achieve strong performance in image classification but incur high computational costs from processing all image tokens. To reduce inference costs in large ViTs without compromising accuracy, we propose TinyDrop, a training-free token dropping framework guided by a lightweight vision model. The guidance model estimates the importance of tokens while performing inference, thereby selectively discarding low-importance tokens if large vit models need to perform attention calculations. The framework operates plug-and-play, requires no architectural modifications, and is compatible with diverse ViT architectures. Evaluations on standard image classification benchmarks demonstrate that our framework reduces FLOPs by up to 80% for ViTs with minimal accuracy degradation, highlighting its generalization capability and practical utility for efficient ViT-based classification.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationSimilar Papers 제목 키워드 기반
HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
High-resolution Vision-Language Models (VLMs) are widely used in multimodal tasks to enhance accuracy by preserving detailed image information. However, these models often generate an excessive number of visual tokens du…
GPULanguage ModellingLarge Language ModelTraining Noise Token Pruning
In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training,…
Self-slimmed Vision Transformer
Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. However, such powerful transformers bring a huge computation burden, because of…
Knowledge DistillationCacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding
Long-form video question answering (VQA) overwhelms current vision-language models (VLMs) because attention and key-value (KV) caches grow with runtime, forcing either expensive inference or near-sighted sliding windows.…
Video Question AnsweringMulti-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model
The vision tokens in multimodal large language models usually exhibit significant spatial and temporal redundancy and take up most of the input tokens, which harms their inference efficiency. To solve this problem, some …
Language ModelingLanguage ModellingLarge Language ModelMultimodal Large Language Model