One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression
Pruning is a core technique for compressing neural networks to improve computational efficiency. This process is typically approached in two ways: one-shot pruning, which involves a single pass of training and pruning, and iterative pruning, where pruning is performed over multiple cycles for potentially finer network refinement. Although iterative pruning has historically seen broader adoption, this preference is often assumed rather than rigorously tested. Our study presents one of the first systematic and comprehensive comparisons of these methods, providing rigorous definitions, benchmarking both across structured and unstructured settings, and applying different pruning criteria and modalities. We find that each method has specific advantages: one-shot pruning proves more effective at lower pruning ratios, while iterative pruning performs better at higher ratios. Building on these findings, we advocate for patience-based pruning and introduce a hybrid approach that can outperform traditional methods in certain scenarios, providing valuable insights for practitioners selecting a pruning strategy tailored to their goals and constraints. Source code is available at https://github.com/janumiko/pruning-benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyModel CompressionSimilar Papers 제목 키워드 기반
Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
Network pruning is one of the most dominant methods for reducing the heavy inference cost of deep neural networks. Existing methods often iteratively prune networks to attain high compression ratio without incurring sign…
image-classificationImage ClassificationKnowledge DistillationModel Compression+1TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning
Large Language Models (LLMs) present significant computational and memory challenges due to their extensive size, making pruning essential for their efficient deployment. Existing one-shot pruning methods often apply uni…
Single-shot Channel Pruning Based on Alternating Direction Method of Multipliers
Channel pruning has been identified as an effective approach to constructing efficient network structures. Its typical pipeline requires iterative pruning and fine-tuning. In this work, we propose a novel single-shot cha…
General ClassificationNetwork PruningOPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization
As Deep Neural Networks (DNNs) usually are overparameterized and have millions of weight parameters, it is challenging to deploy these large DNN models on resource-constrained hardware platforms, e.g., smartphones. Numer…
QuantizationVanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression
The increasing scale of Deep Neural Networks (DNNs) introduces the need for compression techniques such as pruning, quantization, and low-rank decomposition. While these methods are very effective at reducing memory, com…
Model Compression