Compute Or Load KV Cache? Why Not Both?

2024-10-04 · Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao

Recent advancements in Large Language Models (LLMs) have significantly increased context window sizes, enabling sophisticated applications but also introducing substantial computational overheads, particularly computing key-value (KV) cache in the prefill stage. Prefix caching has emerged to save GPU power in this scenario, which saves KV cache at disks and reuse them across multiple queries. However, traditional prefix caching mechanisms often suffer from substantial latency because the speed of loading KV cache from disks to GPU memory is bottlenecked by the throughput of I/O devices. To optimize the latency of long-context prefill, we propose Cake, a novel KV cache loader, which employs a bidirectional parallelized KV cache generation strategy. Upon receiving a prefill task, Cake simultaneously and dynamically loads saved KV cache from prefix cache locations and computes KV cache on local GPUs, maximizing the utilization of available computation and I/O bandwidth resources. Additionally, Cake automatically adapts to diverse system statuses without manual parameter. tuning. In experiments on various prompt datasets, GPUs, and I/O devices, Cake offers up to 68.1% Time To First Token (TTFT) reduction compare with compute-only method and 94.6% TTFT reduction compare with I/O-only method.

📄 PDF Abstract BibTeX arXiv:2410.03065

Code (0)

등록된 구현이 없습니다.

Tasks

GPU

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading

2025-04-16 · Kihyun Kim, Jinwoo Kim, Hyunsun Chung, Myung-Hoon Cha 외

LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes …

GPU Text Summarization

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

2026-09-09 · Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi arxiv

In Retrieval-Augmented Generation (RAG) systems, a large number of retrieved chunks are concatenated to form the input context so that users can receive high-quality responses based on external knowledge. As a result, th…

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

2025-10-08 · Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An 외 arxiv

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache …

Question Answering

Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

2025-02-05 · Shubham Agarwal, Sai Sundaresan, Subrata Mitra, Debabrata Mahapatra 외

Retrieval-Augmented Generation (RAG) is often used with Large Language Models (LLMs) to infuse domain knowledge or user-specific information. In RAG, given a user query, a retriever extracts chunks of relevant text from …

RAG Retrieval Retrieval-augmented Generation

LQoCo: Learning to Optimize Cache Capacity Overloading in Storage Systems

2022-03-21 · Ji Zhang, Xijun Li, Xiyao Zhou, Mingxuan Yuan 외

Cache plays an important role to maintain high and stable performance (i.e. high throughput, low tail latency and throughput jitter) in storage systems. Existing rule-based cache management methods, coupled with engineer…

Management