Zero-Shot Personalized Speech Enhancement through Speaker-Informed Model Selection
This paper presents a novel zero-shot learning approach towards personalized speech enhancement through the use of a sparsely active ensemble model. Optimizing speech denoising systems towards a particular test-time speaker can improve performance and reduce run-time complexity. However, test-time model adaptation may be challenging if collecting data from the test-time speaker is not possible. To this end, we propose using an ensemble model wherein each specialist module denoises noisy utterances from a distinct partition of training set speakers. The gating module inexpensively estimates test-time speaker characteristics in the form of an embedding vector and selects the most appropriate specialist module for denoising the test signal. Grouping the training set speakers into non-overlapping semantically similar groups is non-trivial and ill-defined. To do this, we first train a Siamese network using noisy speech pairs to maximize or minimize the similarity of its output vectors depending on whether the utterances derive from the same speaker or not. Next, we perform k-means clustering on the latent space formed by the averaged embedding vectors per training set speaker. In this way, we designate speaker groups and train specialist modules optimized around partitions of the complete training set. Our experiments show that ensemble models made up of low-capacity specialists can outperform high-capacity generalist models with greater efficiency and improved adaptation towards unseen test-time speakers.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringDenoisingModel SelectionSpeech DenoisingSpeech EnhancementZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement
This paper presents a new challenge that calls for zero-shot text-to-speech (TTS) systems to augment speech data for the downstream task, personalized speech enhancement (PSE), as part of the Generative Data Augmentation…
Data AugmentationSpeech EnhancementSpeech SynthesisSynthetic Data Generation+2Efficient Personalized Speech Enhancement through Self-Supervised Learning
This work presents self-supervised learning methods for developing monaural speaker-specific (i.e., personalized) speech enhancement models. While generalist models must broadly address many speakers, specialist models c…
Few-Shot LearningModel CompressionSelf-Supervised LearningSpeech Enhancement+1Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation
In realistic speech enhancement settings for end-user devices, we often encounter only a few speakers and noise types that tend to reoccur in the specific acoustic environment. We propose a novel personalized speech enha…
DenoisingKnowledge DistillationModel CompressionSpecificity+3Improving Noise Robustness of LLM-based Zero-shot TTS via Discrete Acoustic Token Denoising
Large language model (LLM) based zero-shot text-to-speech (TTS) methods tend to preserve the acoustic environment of the audio prompt, leading to degradation in synthesized speech quality when the audio prompt contains n…
DecoderDenoisingLanguage ModelingLanguage Modelling+4Personalized Speech Enhancement through Self-Supervised Data Augmentation and Purification
Training personalized speech enhancement models is innately a no-shot learning problem due to privacy constraints and limited access to noise-free speech from the target user. If there is an abundance of unlabeled noisy …
Data AugmentationDenoisingPrivacy PreservingSelf-Supervised Learning+1