VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference
This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference
Although vision transformers (ViT) have shown remarkable success in various vision tasks, their computationally expensive self-attention hinder their deployment on resource-constrained devices. Token reduction, which dis…
RetoVLA: Reusing Register Tokens for Spatial Reasoning in Vision-Language-Action Models
Vision-Language-Action (VLA) models have demonstrated robust performance across diverse robotic tasks. However, their high memory and computational demands often limit real-time deployment. While existing model compressi…
Spatial ReasoningModel CompressionTransfer Learning Applied to Computer Vision Problems: Survey on Current Progress, Limitations, and Opportunities
The field of Computer Vision (CV) has faced challenges. Initially, it relied on handcrafted features and rule-based algorithms, resulting in limited accuracy. The introduction of machine learning (ML) has brought progres…
Transfer LearningSimple o3: Towards Interleaved Vision-Language Reasoning
Multimodal Large Language Models (MLLMs) have shown impressive performance on vision-language tasks, but their long Chain-of-Thought (CoT) capabilities in multimodal scenarios remain underexplored. Inspired by OpenAI's o…
Multimodal ReasoningVisual GroundingVisual ReasoningImage CroppingEfficient Long-Horizon Vision-Language-Action Models via Static-Dynamic Disentanglement
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for generalist robotic control. Built upon vision-language model (VLM) architectures, VLAs predict actions conditioned on visual observati…