Self-supervised Learning with Speech Modulation Dropout
We show that training a multi-headed self-attention-based deep network to predict deleted, information-dense 2-8 Hz speech modulations over a 1.5-second section of a speech utterance is an effective way to make machines learn to extract speech modulations using time-domain contextual information. Our work exhibits that, once trained on large volumes of unlabelled data, the outputs of the self-attention layers vary in time with a modulation peak at 4 Hz. These pre-trained layers can be used to initialize parts of an Automatic Speech Recognition system to reduce its reliance on labeled speech data greatly.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionSelf-Supervised Learningspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation
Predicting the altered acoustic frames is an effective way of self-supervised learning for speech representation. However, it is challenging to prevent the pretrained model from overfitting. In this paper, we proposed to…
ClassificationSelf-Supervised LearningSpeaker RecognitionUnsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training
The performance of automatic speech recognition (ASR) systems typically degrades significantly when the training and test data domains are mismatched. In this paper, we show that self-training (ST) combined with an uncer…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Domain AdaptationPseudo Label+4MCR-Data2vec 2.0: Improving Self-supervised Speech Pre-training via Model-level Consistency Regularization
Self-supervised learning (SSL) has shown significant progress in speech processing tasks. However, despite the intrinsic randomness in the Transformer structure, such as dropout variants and layer-drop, improving the mod…
Self-Supervised LearningOn Unsupervised Uncertainty-Driven Speech Pseudo-Label Filtering and Model Calibration
Pseudo-label (PL) filtering forms a crucial part of Self-Training (ST) methods for unsupervised domain adaptation. Dropout-based Uncertainty-driven Self-Training (DUST) proceeds by first training a teacher model on sourc…
Domain AdaptationPseudo LabelPseudo Label FilteringUnsupervised Domain AdaptationSelf-FiLM: Conditioning GANs with self-supervised representations for bandwidth extension based speaker recognition
Speech super-resolution/Bandwidth Extension (BWE) can improve downstream tasks like Automatic Speaker Verification (ASV). We introduce a simple novel technique called Self-FiLM to inject self-supervision into existing BW…
Bandwidth ExtensionSpeaker RecognitionSpeaker VerificationSuper-Resolution+1