홈 › Papers

ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models

2024-06-24 · Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, Mohamed S Abdelfattah

The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at \href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}.

📄 PDF Abstract BibTeX arXiv:2406.16635

Code (1)

abdelfattah-lab/shadow_llm 공식 구현 jax

Tasks

Quantization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

OPT OPT is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. The model uses an AdamW optimizer and weight decay of 0.1. It follows a linear…

Pruning 설명 없음

SVD Contextual Sparsity Predictors for Fast LLM Inference

2026-03-14 · Georgii Serbin, Kirill Koshkin, Zhongao Sun, Anastasiya Bistrigova 외 arxiv

Contextual sparsity is one of the approaches used to reduce computational complexity in the inference process of large language models (LLMs). Existing techniques for efficient LLM inference acceleration based on context…

Code Generation

Safety from Honesty in a Disinterested AI Predictor

2026-06-28 · Yoshua Bengio, Oliver Richardson, Tomáš Gavenčiak, Michael Cohen 외 arxiv

As AI systems become more capable, training procedures that optimize for downstream outcomes risk introducing implicit agency: goal-directed behavior that designers never specified. We present a formal safety argument fo…

Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity

2025-05-20 · Susav Shrestha, Brad Settlemyer, Nikoli Dryden, Narasimha Reddy

Accelerating large language model (LLM) inference is critical for real-world deployments requiring high throughput and low latency. Contextual sparsity, where each token dynamically activates only a small subset of the m…

GPU Large Language Model

Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs

2026-06-09 · Ruixuan Huang, Jinyuan Shi, Hantao Huang, Yifan Huang 외 arxiv

We study dense-to-sparse continual training as a way to construct channel-sparse large language models from dense checkpoints. Starting from a Qwen2.5-8B dense backbone, we continue training at 32K context and introduce …

TokenButler: Token Importance is Predictable

2025-03-10 · Yash Akhauri, Ahmed F AbouElhamayed, YiFei Gao, Chi-Chih Chang 외

Large Language Models (LLMs) rely on the Key-Value (KV) Cache to store token history, enabling efficient decoding of tokens. As the KV-Cache grows, it becomes a major memory and computation bottleneck, however, there is …