Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.
Code (7)
Tasks
General ClassificationRepresentation LearningSentiment AnalysisSentiment ClassificationSpeaker RecognitionSpeech Representation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
In recent times, BERT based transformer models have become an inseparable part of the 'tech stack' of text processing models. Similar progress is being observed in the speech domain with a multitude of models observing s…
AllEmotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge
Speech Self-Supervised Learning (SSL) has demonstrated considerable efficacy in various downstream tasks. Nevertheless, prevailing self-supervised models often overlook the incorporation of emotion-related prior informat…
Representation LearningSelf-Supervised LearningSpeech Representation LearningDefense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learning
High-performance anti-spoofing models for automatic speaker verification (ASV), have been widely used to protect ASV by identifying and filtering spoofing audio that is deliberately generated by text-to-speech, voice con…
Self-Supervised LearningSpeaker Verificationtext-to-speechText to Speech+1Transformer with Bidirectional Decoder for Speech Recognition
Attention-based models have made tremendous progress on end-to-end automatic speech recognition(ASR) recently. However, the conventional transformer-based approaches usually generate the sequence results token by token f…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationDecoder+4Speech representation learning: Learning bidirectional encoders with single-view, multi-view, and multi-task methods
This thesis focuses on representation learning for sequence data over time or space, aiming to improve downstream sequence prediction tasks by using the learned representations. Supervised learning has been the most domi…
MULTI-VIEW LEARNINGRepresentation LearningSelf-Supervised LearningSpeech Representation Learning