Investigating self-supervised learning for speech enhancement and separation
Speech enhancement and separation are two fundamental tasks for robust speech processing. Speech enhancement suppresses background noise while speech separation extracts target speech from interfering speakers. Despite a great number of supervised learning-based enhancement and separation methods having been proposed and achieving good performance, studies on applying self-supervised learning (SSL) to enhancement and separation are limited. In this paper, we evaluate 13 SSL upstream methods on speech enhancement and separation downstream tasks. Our experimental results on Voicebank-DEMAND and Libri2Mix show that some SSL representations consistently outperform baseline features including the short-time Fourier transform (STFT) magnitude and log Mel filterbank (FBANK). Furthermore, we analyze the factors that make existing SSL frameworks difficult to apply to speech enhancement and separation and discuss the representation properties desired for both tasks. Our study is included as the official speech enhancement and separation downstreams for SUPERB.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSpeech EnhancementSpeech SeparationSimilar Papers 제목 키워드 기반
Continual self-training with bootstrapped remixing for speech enhancement
We propose RemixIT, a simple and novel self-supervised training method for speech enhancement. The proposed method is based on a continuously self-training scheme that overcomes limitations from previous studies includin…
Domain AdaptationSpeech EnhancementUnsupervised Domain AdaptationAudio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings
AV-HuBERT, a multi-modal self-supervised learning model, has been shown to be effective for categorical problems such as automatic speech recognition and lip-reading. This suggests that useful audio-visual speech represe…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Lip Readingregression+5ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes t…
Audio-Visual Speech RecognitionResynthesisSpeech Enhancementspeech-recognition+7Deep neural network techniques for monaural speech enhancement: state of the art analysis
Deep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in these domains in task such as machine translation and im…
Art AnalysisImage GenerationMachine TranslationSpeaker Separation+2Improved Speech Enhancement with the Wave-U-Net
We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment. This end-to-end learning method for audio source separatio…
Audio Source SeparationSpeech Enhancementspeech-recognitionSpeech Recognition