ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT

2026-02-17 · Hyunchan Moon, Cheonjun Park, Steven L. Waslander arxiv

Vision Transformers (ViTs) have achieved remarkable success across various vision tasks, yet their deployment is often hindered by prohibitive computational costs. While structured weight pruning and token compression have emerged as promising solutions, they suffer from prolonged retraining and inter-layer dependencies that complicate optimization, respectively. We propose ToaSt, a decoupled framework applying specialized strategies to distinct ViT components. We apply coupled head-wise structured pruning to Multi-Head Self-Attention modules, leveraging attention operation characteristics to enhance robustness. For Feed-Forward Networks (over 60% of FLOPs), we introduce Token Channel Selection (TCS), a training-free method that filters redundant noise channels at inference time. Extensive evaluations across nine diverse models, including DeiT, ViT-MAE, and Swin Transformer, demonstrate that ToaSt achieves superior trade-offs between accuracy and efficiency, consistently outperforming existing baselines. On ViT-MAE-Huge, ToaSt achieves 88.52% accuracy (+1.64%p) with 39.4% FLOPs reduction. ToaSt also transfers effectively to diverse downstream tasks (COCO detection, ADE20K segmentation, CIFAR-100 classification), achieving 52.2 versus 51.9 mAP on COCO. Code: github.com/SHANNonLab-HUFS/ToaSt

📄 PDF Abstract BibTeX arXiv:2602.15720

Code (0)

등록된 구현이 없습니다.

Tokenization with Split Trees

2026-05-21 · Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage, Seth Ebner 외 arxiv

We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily splits each pretoken into a full binary tree…

Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inference

2026-05-19 · Beomseok Kang, Dongwon Jo, Jiwon Song, Donghwee Son 외 arxiv

Vision-Language Models suffer severe KV cache pressure at inference, as a single image often encodes into thousands of tokens. Most existing methods exploit token sparsity through token pruning, but permanently discardin…

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

2026-08-04 · Yaozhi Wen, Jialong Guo, Zhenliang Ni, Han Shu 외 arxiv

While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their de…

Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

2026-07-02 · Xuehui Wang, Xuankun Yang, Wei Shen arxiv

Visual token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries. In thi…

CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transport

2026-08-13 · Peng Ling, Yingda Yin, Lingting Zhu, Weikai Chen 외 arxiv

While 3D Vision-Language Models (3D VLMs) have demonstrated remarkable spatial reasoning capabilities, they suffer from massive visual token counts that create severe computational bottlenecks during inference. Existing …

Spatial Reasoning