Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speech
In recent studies, self-supervised pre-trained models tend to outperform supervised pre-trained models in transfer learning. In particular, self-supervised learning (SSL) of utterance-level speech representation can be used in speech applications that require discriminative representation of consistent attributes within an utterance: speaker, language, emotion, and age. Existing frame-level self-supervised speech representation, e.g., wav2vec, can be used as utterance-level representation with pooling, but the models are usually large. There are also SSL techniques to learn utterance-level representation. One of the most successful is a contrastive method, which requires negative sampling: selecting alternative samples to contrast with the current sample (anchor). However, this does not ensure that all the negative samples belong to classes different from the anchor class without labels. This paper applies a non-contrastive self-supervised method to learn utterance-level embeddings. We adapted DIstillation with NO labels (DINO) from computer vision to speech. Unlike contrastive methods, DINO does not require negative sampling. We compared DINO to x-vector trained in a supervised manner. When transferred to down-stream tasks (speaker verification, speech emotion recognition (SER), and Alzheimer's disease detection), DINO outperformed x-vector. We studied the influence of several aspects during transfer learning such as dividing the fine-tuning process into steps, chunk lengths, or augmentation. During fine-tuning, tuning the last affine layers first and then the whole network surpassed fine-tuning all at once. Using shorter chunk lengths, although they generate more diverse inputs, did not necessarily improve performance, implying speech segments at least with a specific length are required for better performance per application. Augmentation was helpful in SER.
Code (0)
등록된 구현이 없습니다.
Tasks
Alzheimer's Disease DetectionEmotion RecognitionSelf-Supervised LearningSpeaker VerificationSpeech Emotion RecognitionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Utterance Rewriting with Contrastive Learning in Multi-turn Dialogue
Context modeling plays a significant role in building multi-turn dialogue systems. In order to make full use of context information, systems can use Incomplete Utterance Rewriting(IUR) methods to simplify the multi-turn …
Contrastive LearningMulti-Task LearningSentenceUtterance Rewriting with Contrastive Learning in Multi-turn Dialogue
Context modeling plays a significant role in building multi-turn dialogue systems. In order to make full use of context information, systems can use Incomplete Utterance Rewriting(IUR) methods to simplify the multi-turn …
Contrastive LearningMulti-Task LearningSentenceNon-Contrastive Self-Supervised Learning of Utterance-Level Speech Representations
Considering the abundance of unlabeled speech data and the high labeling costs, unsupervised learning methods can be essential for better system development. One of the most successful methods is contrastive self-supervi…
Emotion RecognitionSelf-Supervised LearningSpeaker VerificationTaDSE: Template-aware Dialogue Sentence Embeddings
Learning high quality sentence embeddings from dialogues has drawn increasing attentions as it is essential to solve a variety of dialogue-oriented tasks with low annotation cost. However, directly annotating and gatheri…
Contrastive Learningintent-classificationIntent ClassificationSemantic Compression+6Augmentation adversarial training for self-supervised speaker recognition
The goal of this work is to train robust speaker recognition models without speaker labels. Recent works on unsupervised speaker representations are based on contrastive learning in which they encourage within-utterance …
Contrastive LearningSpeaker Recognition