Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations

2024-07-08 · Bowen Shen, Zheng Lin, Daren Zha, Wei Liu, Jian Luan, Bin Wang, Weiping Wang

Structured pruning fundamentally reduces computational and memory overheads of large language models (LLMs) and offers a feasible solution for end-side LLM deployment. Structurally pruned models remain dense and high-precision, highly compatible with further tuning and compression. However, as the coarse-grained structured pruning poses large damage to the highly interconnected model, achieving a high compression ratio for scaled-up LLMs remains a challenge. In this paper, we introduce a task-agnostic structured pruning approach coupled with a compact Transformer architecture design. The proposed approach, named TransAct, reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules, while preserving the inter-module activations that are sensitive to perturbations. Hence, the LLM is pruned into an intra-module low-rank architecture, significantly reducing weights, KV Cache and attention computation. TransAct is implemented on the LLaMA model and evaluated on downstream benchmarks. Results verify the optimality of our approach at high compression with respect to both efficiency and performance. Further, ablation studies reveal the strength of activation-guided iterative pruning and provide experimental analysis on the redundancy of MHA and MLP modules.

📄 PDF Abstract BibTeX arXiv:2407.05690

Code (1)

sbwww/transact-pruning 공식 구현

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

IntraSlice: Towards High-Performance Structural Pruning with Block-Intra PCA for LLMs

2026-02-02 · Meng Li, Peisong Wang, Yuantian Shao, Qinghao Hu 외 arxiv

Large Language Models (LLMs) achieve strong performance across diverse tasks but face deployment challenges due to their massive size. Structured pruning offers acceleration benefits but leads to significant performance …

Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfe

2025-10-15 · Christophe Roux, Max Zimmer, Alexandre d'Aspremont, Sebastian Pokutta arxiv

Pruning is a common technique to reduce the compute and storage requirements of Neural Networks. While conventional approaches typically retrain the model to recover pruning-induced performance degradation, state-of-the-…

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

2026-03-12 · Xiaoyun Liu, Divya Saxena, Jiannong Cao, Yuqing Zhao 외 arxiv

Structured pruning is widely used to compress large language models (LLMs), yet its effectiveness depends heavily on neuron importance estimation. Most existing methods estimate neuron importance from activation statisti…

DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

2025-08-24 · Haojie Zhang arxiv

LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead…

parameter-efficient fine-tuning Mathematical Reasoning Code Generation

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

2024-11-01 · Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang 외

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with consid…

Mixture-of-Experts