Progressive Multi-Scale Self-Supervised Learning for Speech Recognition
Self-supervised learning (SSL) models have achieved considerable improvements in automatic speech recognition (ASR). In addition, ASR performance could be further improved if the model is dedicated to audio content information learning theoretically. To this end, we propose a progressive multi-scale self-supervised learning (PMS-SSL) method, which uses fine-grained target sets to compute SSL loss at top layer while uses coarse-grained target sets at intermediate layers. Furthermore, PMS-SSL introduces multi-scale structure into multi-head self-attention for better speech representation, which restricts the attention area into a large scope at higher layers while restricts the attention area into a small scope at lower layers. Experiments on Librispeech dataset indicate the effectiveness of our proposed method. Compared with HuBERT, PMS-SSL achieves 13.7% / 12.7% relative WER reduction on test other evaluation subsets respectively when fine-tuned on 10hours / 100hours subsets.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Self-Supervised Learningspeech-recognitionSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Progressive Residual Extraction based Pre-training for Speech Representation Learning
Self-supervised learning (SSL) has garnered significant attention in speech processing, excelling in linguistic tasks such as speech recognition. However, jointly improving the performance of pre-trained models on variou…
Emotion RecognitionRepresentation LearningSelf-Supervised LearningSpeaker Identification+5Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM
Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation, a critical challenge in multi-party speech processing. This work proposes a speaker-aware progressive OSD mod…
Action DetectionActivity DetectionSelf-Supervised LearningProgressive Video Summarization via Multimodal Self-supervised Learning
Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-…
Self-Supervised LearningSupervised Video SummarizationVideo ClassificationVideo SummarizationES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations
We propose a novel strategy ES3 for self-supervised learning of robust audio-visual speech representations from unlabeled talking face videos. While many recent approaches for this task primarily rely on guiding the …
Audio-Visual Speech RecognitionLipreadingSelf-Supervised LearningSpeech RecognitionMAST: Multiscale Audio Spectrogram Transformers
We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram…
Audio ClassificationKeyword SpottingSelf-Supervised Learning