ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

2026-05-16 · Yu Zhu, Aditya Dhakal, Yunming Xiao, Dejan Milojicic, Gustavo Alonso arxiv

Prefix KV caching has become a key mechanism in LLM serving: it reduces time to first token (TTFT) by avoiding redundant computation across requests that share a prefix (i.e., the system prompt). However, the accumulated KV cache is often larger than what GPU memory and local DRAM can hold. To preserve latency, current systems keep the KV cache in remote DRAM pools, increasing serving-cluster size and cost. In this paper, we explore a different approach: storing the KV cache in S3-compatible object storage so that capacity is no longer the constraint, while minimizing the impact on TTFT. We propose ObjectCache, which co-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it, overlapping data transfer with compute across concurrent requests. We prototype ObjectCache on a 100 Gbps RoCE cluster with NIXL (an inference library that abstracts storage and memory), Ceph RGW (an Object Gateway for clusters), and DAOS (an open source storage system). For 64K contexts, common in today's systems, ObjectCache adds only 5.6\% latency over local DRAM; for 4K contexts, where less compute is available to mask transfer, ObjectCache adds 56--75\,ms over the optimal local layerwise baseline. Under shared bandwidth caps, our scheduler reduces added TTFT by 1.2--1.8x compared with equal bandwidth sharing.

📄 PDF Abstract BibTeX arXiv:2605.22850

Code (0)

등록된 구현이 없습니다.

Cache Mechanism for Agent RAG Systems

2025-11-04 · Shuhang Lin, Zhencan Peng, Lingyao Li, Xiao Lin 외 arxiv

Recent advances in Large Language Model (LLM)-based agents have been propelled by Retrieval-Augmented Generation (RAG), which grants the models access to vast external knowledge bases. Despite RAG's success in improving …

RLCache: Automated Cache Management Using Reinforcement Learning

2019-09-30 · Sami Alabed

This study investigates the use of reinforcement learning to guide a general purpose cache manager decisions. Cache managers directly impact the overall performance of computer systems. They govern decisions about which …

Management reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference

2026-01-20 · Zhiyuan Shi, Qibo Qiu, Feng Xue, Zhonglin Jiang 외 arxiv

The linear memory growth of the KV cache poses a significant bottleneck for LLM inference in long-context tasks. Existing static compression methods often fail to preserve globally important information. Although recent …

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

2024-05-26 · Jiayi Yao, Hanchen Li, YuHan Liu, Siddhant Ray 외

Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use…

Language Modeling Language Modelling Large Language Model RAG

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

2024-07-01 · Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu 외

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now b…

GPU Quantization Retrieval