ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models
In Vision-Language Models (VLMs), processing a massive number of visual tokens incurs prohibitive computational overhead. While recent training-aware pruning methods attempt to selectively discard redundant tokens, they …
Reinforcement LearningDSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
Budgeted pruning is the problem of pruning under resource constraints. In budgeted pruning, how to distribute the resources across layers (i.e., sparsity allocation) is the key problem. Traditional methods solve it by di…
Review of a Heaviside step sequence function and the recursive Heaviside step sequence function for modeling human mental state
In this paper, we define a novel recursive Heaviside step sequence function and demonstrate its applicability to modeling human mental states such as thought processes, memory recall, and forgetfulness. By extending the …
MaskConvNet: Training Efficient ConvNets from Scratch via Budget-constrained Filter Pruning
In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a Con…
Network PruningTopology-Aware Revival for Efficient Sparse Training
Static sparse training is a promising route to efficient learning by committing to a fixed mask pattern, yet the constrained structure reduces robustness. Early pruning decisions can lock the network into a brittle struc…
Reinforcement Learning