paper-with-me

홈 › Papers

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

2026-06-09 · Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari arxiv

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

📄 PDF Abstract BibTeX arXiv:2606.10445

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

2025-12-15 · Mohammad Mozaffari, Samuel Kushnir, Maryam Mehri Dehnavi, Amir Yazdanbakhsh arxiv

Post-training model pruning is a promising solution, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are compu…

ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning

2026-03-06 · Mingluo Su, Huan Wang arxiv

Pruning is widely recognized as an effective method for reducing the parameters of large language models (LLMs), potentially leading to more efficient deployment and inference. One classic and prominent path of LLM one-s…

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism

2024-08-20 · Guanchen Li, Xiandong Zhao, Lian Liu, Zeping Li 외

Pre-trained language models (PLMs) are engineered to be robust in contextual understanding and exhibit outstanding performance in various natural language processing tasks. However, their considerable size incurs signifi…

SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask

2026-05-07 · Liu Hanzuo, Chaofan Lin, Weixuan Sun, Yulong Wang 외 arxiv

Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers from substantial quality degradation due…

Rethinking Token Pruning for Historical Screenshots in GUI Visual Agents: Semantic, Spatial, and Temporal Perspectives

2026-03-27 · Daiqiang Li, Zihao Pan, Zeyu Zhang, Ronghao Chen 외 arxiv

In recent years, GUI visual agents built upon Multimodal Large Language Models (MLLMs) have demonstrated strong potential in navigation tasks. However, high-resolution GUI screenshots produce a large number of visual tok…