Distilling Knowledge Using Parallel Data for Far-field Speech Recognition
In order to improve the performance for far-field speech recognition, this paper proposes to distill knowledge from the close-talking model to the far-field model using parallel data. The close-talking model is called the teacher model. The far-field model is called the student model. The student model is trained to imitate the output distributions of the teacher model. This constraint can be realized by minimizing the Kullback-Leibler (KL) divergence between the output distribution of the student model and the teacher model. Experimental results on AMI corpus show that the best student model achieves up to 4.7% absolute word error rate (WER) reduction when compared with the conventionally-trained baseline models.
Code (0)
등록된 구현이 없습니다.
Tasks
speech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers
Lip reading has witnessed unparalleled development in recent years thanks to deep learning and the availability of large-scale datasets. Despite the encouraging results achieved, the performance of lip reading, unfortuna…
Knowledge DistillationLipreadingLip Readingspeech-recognition+1Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation
Much research effort is being applied to the task of compressing the knowledge of self-supervised models, which are powerful, yet large and memory consuming. In this work, we show that the original method of knowledge di…
Automatic Speech RecognitionKnowledge Distillationspeech-recognitionSpeech RecognitionDistilling a Pretrained Language Model to a Multilingual ASR Model
Multilingual speech data often suffer from long-tailed language distribution, resulting in performance degradation. However, multilingual text data is much easier to obtain, yielding a more useful general language model.…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+3A Comprehensive Survey on Knowledge Distillation of Diffusion Models
Diffusion Models (DMs), also referred to as score-based diffusion models, utilize neural networks to specify score functions. Unlike most other probabilistic models, DMs directly model the score functions, which makes th…
Knowledge DistillationSurveyWhispering Context: Distilling Syntax and Semantics for Long Speech Transcripts
ASR systems often struggle with maintaining syntactic and semantic accuracy in long audio transcripts, impacting tasks like Named Entity Recognition (NER), capitalization, and punctuation. We propose a novel approach tha…