Pretext Tasks selection for multitask self-supervised speech representation learning
Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experiments conducted on automatic speech recognition, speaker and emotion recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.
Code (1)
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion RecognitionRepresentation LearningSelf-Supervised LearningSpeaker Recognitionspeech-recognitionSpeech RecognitionSpeech Representation LearningSimilar Papers 제목 키워드 기반
Quantitative Estimation of Target Task Performance from Unsupervised Pretext Task in Semi/Self-Supervised Learning
The effectiveness of unlabeled data in Semi/Self-Supervised Learning (SSL) depends on appropriate assumptions for specific scenarios, thereby enabling the selection of beneficial unsupervised pretext tasks. However, exis…
Self-Supervised LearningSelf-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes
Self-supervised learning has emerged as a powerful tool for pretraining deep networks on unlabeled data, prior to transfer learning of target tasks with limited annotation. The relevance between the pretraining pretext a…
AnatomyMedical Image AnalysisRepresentation LearningSelf-Supervised Learning+2Self-Supervised Learning of Pretext-Invariant Representations
The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many …
Contrastive Learningobject-detectionObject DetectionRepresentation Learning+3Automated Self-Supervised Learning for Graphs
Graph self-supervised learning has gained increasing attention due to its capacity to learn expressive node representations. Many pretext tasks, or loss functions have been designed from distinct perspectives. However, w…
ClusteringNode ClassificationNode ClusteringSelf-Supervised LearningSelf-Supervised Learning for Group Equivariant Neural Networks
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with th…
Self-Supervised Learning