When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving
Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficiency but degrades the other. Existing systems optimize them in isolation, overlooking that the optimal EMB-KV allocation ratio can shift by up to 0.35 across workload regimes, leaving 20-30\% latency improvement unrealized. While online reallocation is required to close this gap, naive approaches introduce H2D refill traffic on the critical path, causing P99 SLO violations. To address this, we present RACER, which jointly manages HBM allocation and request routing at runtime through two key components: (1) Adaptive Memory Allocation, a three-layer PPO-based controller (frozen base policy, online residual adapter, and burst-aware recovery controller) that achieves $32\,\mathrm{μs}$ decision latency while staying within 0.024-0.029 of the offline-optimal ratio; and (2) EMB-KV-Aware Scheduling, which routes requests by jointly considering KV residency, embedding locality, and node load to avoid routing inefficiencies under heterogeneous allocations. Evaluations on three production-scale datasets over a 32-node A100 cluster show that RACER reduces P99 latency by 24-38\% over the best static policy and achieves 93.5-99.6\% SLO satisfaction across Steady, Trend, and Burst workloads, significantly outperforming state-of-the-art baselines without sacrificing throughput.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems
Deep learning-based models are utilized to achieve state-of-the-art performance for recommendation systems. A key challenge for these models is to work with millions of categorical classes or tokens. The standard approac…
Recommendation SystemsPaged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
Large Language Models (LLMs) encounter severe memory inefficiencies during long-context inference due to conventional handling of key-value (KV) caches. In this work, we introduce a novel integration of PagedAttention wi…
GPUReinforcement Learning for Dynamic Memory Allocation
In recent years, reinforcement learning (RL) has gained popularity and has been applied to a wide range of tasks. One such popular domain where RL has been effective is resource management problems in systems. We look to…
Managementreinforcement-learningReinforcement LearningReinforcement Learning (RL)Structured Token Retention and Computational Memory Paths in Large Language Models
Memory retention mechanisms play a central role in determining the efficiency of computational architectures designed for processing extended sequences. Conventional methods for token management often impose fixed retent…
Information RetrievalOptimal Auction Design for Dynamic Stochastic Environments: Myerson Meets Naor
Allocation of goods and services often involves both stochastic supply and stochastic demand. Motivated by applications such as cloud computing, gig platforms, and blockchain auctions, we study the design of optimal sell…
Cloud Computing