Probing Self-supervised Learning Models with Target Speech Extraction

2024-02-17 · Junyi Peng, Marc Delcroix, Tsubasa Ochiai, Oldrich Plchot, Takanori Ashihara, Shoko Araki, Jan Cernocky

Large-scale pre-trained self-supervised learning (SSL) models have shown remarkable advancements in speech-related tasks. However, the utilization of these models in complex multi-talker scenarios, such as extracting a target speaker in a mixture, is yet to be fully evaluated. In this paper, we introduce target speech extraction (TSE) as a novel downstream task to evaluate the feature extraction capabilities of pre-trained SSL models. TSE uniquely requires both speaker identification and speech separation, distinguishing it from other tasks in the Speech processing Universal PERformance Benchmark (SUPERB) evaluation. Specifically, we propose a TSE downstream model composed of two lightweight task-oriented modules based on the same frozen SSL model. One module functions as a speaker encoder to obtain target speaker information from an enrollment speech, while the other estimates the target speaker's mask to extract its speech from the mixture. Experimental results on the Libri2mix datasets reveal the relevance of the TSE downstream task to probe SSL models, as its performance cannot be simply deduced from other related tasks such as speaker verification and separation.

📄 PDF Abstract BibTeX arXiv:2402.13200

Code (0)

등록된 구현이 없습니다.

Tasks

Self-Supervised Learning Speaker Identification Speaker Verification Speech Extraction Speech Separation

Probing in the Wild: A Case Study of Self-Supervised Speech Representations on Mandarin Sub-dialects with Unsupervised Articulatory Analysis

2026-06-24 · Shu Shang, Fuliang Weng, Zeqian Hu, Yaqian Zhou arxiv

While self-supervised speech models have achieved strong performance across speech tasks, relatively little is known about how their internal phonetic representations behave under fine-grained dialect variation. Existing…

Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective

2024-01-16 · Alexander H. Liu, Sung-Lin Yeh, James Glass

Existing studies on self-supervised speech representation learning have focused on developing new training methods and applying pre-trained models for different applications. However, the quality of these models is often…

Representation Learning Self-Supervised Learning speech-recognition Speech Recognition+1

Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction

2023-12-16 · Zhaoxi Mu, Xinyu Yang, Sining Sun, Qing Yang

Speech signals are inherently complex as they encompass both global acoustic characteristics and local semantic information. However, in the task of target speech extraction, certain elements of global and local semantic…

Disentanglement Representation Learning Speech Extraction

Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads

2023-08-28 · Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid 외

Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of compreh…

Benchmarking Self-Supervised Learning

Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation

2020-05-18 · Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh 외

For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order t…

Self-Supervised Learning Speaker Identification