Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
In this paper, we investigate distillation and pruning methods to reduce model size for non-intrusive speech quality assessment based on self-supervised representations. Our experiments build on XLS-R-SQA, a speech quality assessment model using wav2vec 2.0 XLS-R embeddings. We retrain this model on a large compilation of mean opinion score datasets, encompassing over 100,000 labeled clips. For distillation, using this model as a teacher, we generate pseudo-labels on unlabeled degraded speech signals and train student models of varying sizes. For pruning, we use a data-driven strategy. While data-driven pruning performs better at larger model sizes, distillation on unlabeled data is more effective for smaller model sizes. Distillation can halve the gap between the baseline's correlation with ground-truth MOS labels and that of the XLS-R-based teacher model, while reducing model size by two orders of magnitude compared to the teacher model.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deep Neural Compression Via Concurrent Pruning and Self-Distillation
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational simila…
Knowledge DistillationLanguage ModelingLanguage ModellingSelf-Distilled Pruning Of Neural Networks
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational simila…
Knowledge DistillationLanguage ModelingLanguage ModellingSelf-Distilled Pruning of Neural Networks
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational simila…
Knowledge DistillationLanguage ModelingLanguage ModellingAligned Weight Regularizers for Pruning Pretrained Neural Networks
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational simila…
Knowledge DistillationLanguage ModelingLanguage ModellingDPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
Self-supervised learning (SSL) has achieved notable success in many speech processing tasks, but the large model size and heavy computational cost hinder the deployment. Knowledge distillation trains a small student mode…
Knowledge DistillationSelf-Supervised Learning