Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

2024-03-14 · Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti

Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we propose Dynamic Memory Compression (DMC), a method for online key-value cache compression at inference time. Most importantly, the model learns to apply different compression ratios in different heads and layers. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to 7x throughput increase during auto-regressive inference on an NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible percentage of the original data without adding any extra parameters. DMC preserves the original downstream performance with up to 4x cache compression, outperforming up-trained grouped-query attention (GQA) and key-value eviction policies (H$_2$O, TOVA). GQA and DMC can be even combined to obtain compounded gains. Hence, DMC can serve as a drop-in replacement for KV caching in existing LLMs to fit longer contexts and larger batches within any given memory budget.

📄 PDF Abstract BibTeX arXiv:2403.09636

Code (1)

NVIDIA/Megatron-LM pytorch

Tasks

GPU

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Grouped-query attention 설명 없음

BackSlash: Rate Constrained Optimized Training of Large Language Models

2025-04-23 · Jun Wu, Jiangtao Wen, Yuxing Han

The rapid advancement of large-language models (LLMs) has driven extensive research into parameter compression after training has been completed, yet compression during the training phase remains largely unexplored. In t…

Auxiliary-predicted Compress Memory Model(ApCM Model): A Neural Memory Storage Model Based on Invertible Compression and Learnable Prediction

2026-01-09 · Weinuo Ou arxiv

Current large language models (LLMs) generally lack an effective runtime memory mechanism,making it difficult to adapt to dynamic and personalized interaction requirements. To address this issue, this paper proposes a no…

GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models

2026-01-08 · Maanas Taneja, Purab Shingvi arxiv

The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We …

Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting

2023-11-22 · Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu 외

Incorporating factual knowledge in knowledge graph is regarded as a promising approach for mitigating the hallucination of large language models (LLMs). Existing methods usually only use the user's input to query the kno…

Hallucination Language Modeling Language Modelling Large Language Model

FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression

2026-04-22 · Ye Qiao, Yian Wang, Zhiheng Chen, Hyoukjun Kwon 외 arxiv

Compressing large language models (LLMs) for deployment on commodity GPUs remains challenging: conventional scalar quantization is limited to fixed bit-widths (e.g., 8/4/3-bit), offers only a few discrete compression poi…