Representation Shift: Unifying Token Compression with FlashAttention

2025-08-01 · Joonmyung Choi, Sanghyeok Lee, Byungoh Ko, Eunseo Kim, Jihyung Kil, Hyunwoo J. Kim arxiv

Transformers have demonstrated remarkable success across vision, language, and video. Yet, increasing task complexity has led to larger models and more tokens, raising the quadratic cost of self-attention and the overhead of GPU memory access. To reduce the computation cost of self-attention, prior work has proposed token compression techniques that drop redundant or less informative tokens. Meanwhile, fused attention kernels such as FlashAttention have been developed to alleviate memory overhead by avoiding attention map construction and its associated I/O to HBM. This, however, makes it incompatible with most training-free token compression methods, which rely on attention maps to determine token importance. Here, we propose Representation Shift, a training-free, model-agnostic metric that measures the degree of change in each token's representation. This seamlessly integrates token compression with FlashAttention, without attention maps or retraining. Our method further generalizes beyond Transformers to CNNs and state space models. Extensive experiments show that Representation Shift enables effective token compression compatible with FlashAttention, yielding significant speedups of up to 5.5% and 4.4% in video-text retrieval and video QA, respectively. Code is available at https://github.com/mlvlab/Representation-Shift.

📄 PDF Abstract BibTeX arXiv:2508.00367

Code (0)

등록된 구현이 없습니다.

Tasks

Video-Text Retrieval

Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models

2026-02-13 · Omer Faruk Deniz, Ruiyu Mao, Ruochen Li, Yapeng Tian 외 arxiv

Multimodal Large Language Models (MLLMs) incur significant computational cost from processing numerous vision tokens through all LLM layers. Prior pruning methods operate either before the LLM, limiting generality due to…

ApET: Approximation-Error Guided Token Compression for Efficient VLMs

2026-02-23 · Qiankun Ma, Ziyao Zhang, Haofei Wang, Jie Chen 외 arxiv

Recent Vision-Language Models (VLMs) have demonstrated remarkable multimodal understanding capabilities, yet the redundant visual tokens incur prohibitive computational overhead and degrade inference efficiency. Prior st…

OjaKV: Context-Aware Online Low-Rank KV Cache Compression

2025-09-25 · Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan 외 arxiv

The expanding long-context capabilities of large language models are constrained by a significant memory bottleneck: the key-value (KV) cache required for autoregressive generation. This bottleneck is substantial; for in…

Towards Lossless Ultimate Vision Token Compression for VLMs

2025-12-09 · Dehua Zheng, Mouxiao Huang, Borui Jiang, Hailin Hu 외 arxiv

Visual language models encounter challenges in computational efficiency and latency, primarily due to the substantial redundancy in the token representations of high-resolution images and videos. Current attention/simila…

Computational Efficiency

Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models

2026-07-12 · Zhaoyang Li, Yanjun Li, Wangkai Li, Yujia Chen 외 arxiv

Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information,…