Adaptive Pruning of Pretrained Transformer via Differential Inclusions
Large transformers have demonstrated remarkable success, making it necessary to compress these models to reduce inference costs while preserving their perfor-mance. Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio, which results in high computational costs. In contrast, we propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. Therefore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization. In this paper, we introduce such an effective pruning method, termed SPP (Solution Path Pruning). To achieve effective pruning, we segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low-rank compression to these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP.
Code (0)
등록된 구현이 없습니다.
Tasks
Low-rank compressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Set-Valued Risk Measures as Backward Stochastic Difference Inclusions and Equations
Scalar dynamic risk measures for univariate positions in continuous time are commonly represented as backward stochastic differential equations. In the multivariate setting, dynamic risk measures have been defined and st…
Split LBI for Deep Learning: Structural Sparsity via Differential Inclusion Paths
Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error. However, compressive networks are desired in…
A fast algorithm for solving the lasso problem exactly without homotopy using differential inclusions
We prove in this work that the well-known lasso problem can be solved exactly without homotopy using novel differential inclusions techniques. Specifically, we show that a selection principle from the theory of different…
Finite-Time Convergence of Continuous-Time Optimization Algorithms via Differential Inclusions
In this paper, we propose two discontinuous dynamical systems in continuous time with guaranteed prescribed finite-time local convergence to strict local minima of a given cost function. Our approach consists of exploiti…
Movement Pruning: Adaptive Sparsity by Fine-Tuning
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural languag…
Network PruningTransfer Learning