Structured Pruning Learns Compact and Accurate Models
The growing size of neural language models has led to increased attention in model compression. Pruning methods start from a large model and gradually remove model weights---they can significantly reduce the model size but hardly achieve impressive runtime efficiency. On the other hand, distillation methods start from a shallower, compact model and can obtain large speedups---however, they are costly to train on large amounts of unlabeled data. In this work, we show that structured pruning can match the distillation counterparts in both latency ($>$10$\times$) and accuracy ($>$92\%) and result in highly compact and efficient subnetworks. Unlike distillation, our task-specific pruning approach, {\ours}, does not need to pre-specify the model architecture nor rely on unlabeled data. Our solution is to jointly prune layers and sub-modules such as heads and hidden units in Transformer models through $l_0$ regularization while ensuring that the resulting model is parallelizable. We also propose a layerwise distillation approach to further guide pruning. Finally, our pruned structures reveal interesting patterns---for example, more than 70\% of feed-forward and 50\% of self-attention layers can be easily pruned, while the first and last 1-2 layers are likely to remain for highly compressed models.
Code (0)
등록된 구현이 없습니다.
Tasks
Model CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Structured Pruning Learns Compact and Accurate Models
The growing size of neural language models has led to increased attention in model compression. The two predominant approaches are pruning, which gradually removes weights from a pre-trained model, and distillation, whic…
Model CompressionPivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
The rapid growth of Large Language Models has driven demand for effective model compression techniques to reduce memory and computation costs. Low-rank pruning has gained attention for its tensor coherence and GPU compat…
GPUModel CompressionCompresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
Despite the remarkable success of Large Language Models (LLMs), the massive size poses significant deployment challenges, particularly on resource-constrained hardware. While existing LLM compression methods focus on qua…
MMLUNatural Language UnderstandingReading ComprehensionRobust Binary Models by Pruning Randomly-initialized Networks
Robustness to adversarial attacks was shown to require a larger model capacity, and thus a larger memory footprint. In this paper, we introduce an approach to obtain robust yet compact models by pruning randomly-initiali…
Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
Self-supervised learning (SSL) models like WavLM can be effectively utilized when building speaker diarization systems but are often large and slow, limiting their use in resource constrained scenarios. Previous studies …
GPUKnowledge DistillationSelf-Supervised Learningspeaker-diarization+1