Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment. While numerous compression techniques have been proposed, inconsistent evaluation metrics make it difficult to compare their practical effectiveness. In this work, we conduct a comprehensive study of four common compression methods, including weight pruning, head pruning, low-rank approximation, and knowledge distillation on self-supervised speech Transformers. We evaluate each method under three key metrics: parameter count, multiply-accumulate operations, and real-time factor. Results show that each method offers distinct advantages. In addition, we contextualize recent compression techniques, comparing DistilHuBERT, FitHuBERT, LightHuBERT, ARMHuBERT, and STaRHuBERT under the same framework, offering practical guidance on compression for deployment.
Code (1)
Tasks
Knowledge DistillationModel CompressionSelf-Supervised LearningSimilar Papers 제목 키워드 기반
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
Compressing large language models (LLMs), often consisting of billions of parameters, provides faster inference, smaller memory footprints, and enables local deployment. Two standard compression techniques are pruning an…
DecoderQuantizationData-Free Knowledge Distillation for Deep Neural Networks
Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches…
Data-free Knowledge DistillationKnowledge DistillationModel CompressionRefBERT: Compressing BERT by Referencing to Pre-computed Representations
Recently developed large pre-trained language models, e.g., BERT, have achieved remarkable performance in many downstream natural language processing applications. These pre-trained language models often contain hundreds…
Knowledge DistillationCompRess: Self-Supervised Learning by Compressing Representations
Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between sup…
Linear evaluationModel CompressionSelf-Supervised Image ClassificationDistilling Neural Networks for Greener and Faster Dependency Parsing
The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression techniqu…
CPUDependency ParsingGPU