DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embeddings to yield a machine-discovered phone inventory, and finally uses the discretized tokens to guide a student network. We show that DinoSR surpasses previous state-of-the-art performance in several downstream tasks, and provide a detailed analysis of the model and the learned discrete units.
Code (1)
Tasks
ClusteringLanguage ModelingLanguage ModellingMasked Language ModelingOnline ClusteringRepresentation LearningSpeech Representation LearningSimilar Papers 제목 키워드 기반
SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representa…
Representation LearningOnline ClusteringLeave No One Behind: Online Self-Supervised Self-Distillation for Sequential Recommendation
Sequential recommendation methods play a pivotal role in modern recommendation systems. A key challenge lies in accurately modeling user preferences in the face of data sparsity. To tackle this challenge, recent methods …
ClusteringContrastive LearningOnline ClusteringRecommendation Systems+2Domain-Agnostic Clustering with Self-Distillation
Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmenta…
ClusteringData AugmentationDeep ClusteringKnowledge Distillation+2Deep Clustering with Diffused Sampling and Hardness-aware Self-distillation
Deep clustering has gained significant attention due to its capability in learning clustering-friendly representations without labeled data. However, previous deep clustering methods tend to treat all samples equally, wh…
ClusteringContrastive LearningDeep ClusteringKnowledge Distillation+2Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning
Speaker representation learning is crucial for voice recognition systems, with recent advances in self-supervised approaches reducing dependency on labeled data. Current two-stage iterative frameworks, while effective, s…
ClusteringKnowledge DistillationOnline ClusteringPseudo Label+1