Improving generalizability of distilled self-supervised speech processing models under distorted settings
Self-supervised learned (SSL) speech pre-trained models perform well across various speech processing tasks. Distilled versions of SSL models have been developed to match the needs of on-device speech applications. Though having similar performance as original SSL models, distilled counterparts suffer from performance degradation even more than their original versions in distorted environments. This paper proposes to apply Cross-Distortion Mapping and Domain Adversarial Training to SSL models during knowledge distillation to alleviate the performance gap caused by the domain mismatch problem. Results show consistent performance improvements under both in- and out-of-domain distorted setups for different downstream tasks while keeping efficient model size.
Code (1)
Tasks
Knowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Ensemble knowledge distillation of self-supervised speech models
Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we per…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion RecognitionKnowledge Distillation+4Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
Speech representation learning plays a vital role in speech processing. Among them, self-supervised learning (SSL) has become an important research direction. It has been shown that an SSL pretraining model can achieve e…
Multi-Task LearningRepresentation LearningSelf-Supervised LearningSpeech Representation LearningSUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency. The challenge builds upon the SUPERB benchmark and impleme…
Audio GenerationRepresentation LearningSelf-Supervised LearningSpeaker Recognition+1Lightweight feature encoder for wake-up word detection based on self-supervised speech representation
Self-supervised learning method that provides generalized speech representations has recently received increasing attention. Wav2vec 2.0 is the most famous example, showing remarkable performance in numerous downstream s…
Dimensionality ReductionSelf-Supervised LearningSKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
Self-supervised learning (SSL) has achieved remarkable success across various speech-processing tasks. To enhance its efficiency, previous works often leverage the use of compression techniques. A notable recent attempt …
Knowledge DistillationSelf-Supervised Learning