SlimGPT: Layer-wise Structured Pruning for Large Language Models

2024-12-24 · Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu

Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

📄 PDF Abstract BibTeX arXiv:2412.18110

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Pruning 설명 없음

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

Adaptive Signal Resuscitation: Channel-wise Post-Pruning Repair for Sparse Vision Networks

2026-05-20 · Qishi Zhan, Ziheng Chen, Minxuan Hu arxiv

One-shot magnitude pruning can cause severe accuracy collapse in the high-sparsity regime, even when the pruning mask preserves the largest weights. We argue that this failure reflects a granularity mismatch in post-prun…

StructPrune: Structured Global Pruning asymptotics with $\mathcal{O}(\sqrt{N})$ GPU Memory

2025-09-25 · Xinyuan Song, Guangji Bai, Liang Zhao arxiv

Pruning is critical for scaling large language models (LLMs). Global pruning achieves strong performance but requires $\mathcal{O}(N)$ memory, which is infeasible for billion-parameter models. Local pruning reduces GPU m…

LaCo: Large Language Model Pruning via Layer Collapse

2024-02-17 · Yifei Yang, Zouying Cao, Hai Zhao

Large language models (LLMs) based on transformer are witnessing a notable trend of size expansion, which brings considerable costs to both model training and inference. However, existing methods such as model quantizati…

Knowledge Distillation Language Modeling Language Modelling Large Language Model+2

ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models

2023-10-04 · Yi-Lin Sung, Jaehong Yoon, Mohit Bansal

Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities, achieving remarkable advancements on various multimodal downstream tasks. However, …

Model Compression

Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

2026-03-25 · Jimyung Hong, Jaehyung Kim arxiv

Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dim…