Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules

2026-05-27 · Karim Galliamov, Rochelle Choenni, Ivan Titov arxiv

LLMs encode both general capabilities and domain-specific knowledge in a single set of parameters. We ask whether this capacity can be reorganized: keeping broadly useful computation in a shared backbone, while moving specialized knowledge into external memory modules. We propose \emph{knowledge offloading} (KOFF), a framework for decomposing a pretrained LLM into a sparse shared backbone and domain-specific memories. Starting from a frozen base model, we jointly learn a structured pruning mask and lightweight recovery modules, implemented as LoRA adapters and learned key-value caches. Across Llama and Qwen models from 3B to 8B, we find that non-trivial capacity can be moved out of the shared backbone without a large loss in model ability. At around 12\% global sparsity, KOFF preserves much of the unpruned model's performance, while pruning the same frozen model without memories degrades sharply. Ablations show that LoRA and learned KV memories are complementary, and specialization analyses suggest that the learned decomposition is meaningful: language-specific neurons are preferentially removed while language-general neurons largely remain in the backbone. These results suggest that knowledge can be reallocated between a shared core and swappable external memories.

📄 PDF Abstract BibTeX arXiv:2605.29075

Code (0)

등록된 구현이 없습니다.

NOSA: Native and Offloadable Sparse Attention

2025-10-15 · Yuxiang Huang, Pengjie Wang, Jicheng Han, Weilin Zhao 외 arxiv

Decoding throughput improvements from larger inference batches are limited by GPU memory, which is largely consumed by the key-value (KV) cache. Prior training-free KV cache offloading alleviates this by keeping redundan…

Practical offloading for fine-tuning LLM on commodity GPU via learned sparse projectors

2024-06-14 · Siyuan Chen, Zhuofeng Wang, Zelong Guan, Yudong Liu 외

Fine-tuning large language models (LLMs) requires significant memory, often exceeding the capacity of a single GPU. A common solution to this memory challenge is offloading compute and data from the GPU to the CPU. Howev…

CPU GPU

HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

2026-06-19 · Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee 외 arxiv

Diffusion LLMs (dLLMs) improve GPU utilization over autoregressive decoding by generating multiple tokens per forward pass, but their KV cache still grows linearly with context, limiting throughput at long contexts. KV c…

Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning

2026-01-24 · Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari arxiv

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typicall…

parameter-efficient fine-tuning Continual Learning

Fast Inference of Mixture-of-Experts Language Models with Offloading

2023-12-28 · Artyom Eliseev, Denis Mazur

With the widespread adoption of Large Language Models (LLMs), many deep learning practitioners are looking for strategies of running these models more efficiently. One such strategy is to use sparse Mixture-of-Experts (M…

Mixture-of-Experts Quantization