Speaker Group Encoding in Self-supervised Speech Recognition Models
We investigate what self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs). We examine several states of S3Ms: pretrained, finetuned on speaker identification (SID), finetuned on automatic speech recognition (ASR), and ASR-finetuned using a fairness enhancing algorithm. We find that S3Ms encode information about several speaker group categories (SGCs), including their gender, age, dialect, ethnicity, and whether they are a native speaker. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature. On the other hand, finetuning for ASR discards phonetically variant speaker group information (SGI) but retains semantically variant SGI. We find that ASR algorithms designed for fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs. We discuss how SGI is encoded by each layer, and identify subdimensions of embeddings responsible for encoding different SGCs. Finally, we discuss how our findings could be beneficial in designing fairer ASR algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Speaker IdentificationSpeech RecognitionSimilar Papers 제목 키워드 기반
Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech models
Analyses of self-supervised speech models have begun to reveal where and how they represent different types of information. However, almost all analyses have focused on English. Here, we examine how wav2vec2 models train…
Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction
Speech signals are inherently complex as they encompass both global acoustic characteristics and local semantic information. However, in the task of target speech extraction, certain elements of global and local semantic…
DisentanglementRepresentation LearningSpeech ExtractionAn Unsupervised Speaker Clustering Technique based on SOM and I-vectors for Speech Recognition Systems
In this paper, we introduce an enhancement for speech recognition systems using an unsupervised speaker clustering technique. The proposed technique is mainly based on I-vectors and Self-Organizing Map Neural Network(SOM…
Automatic Speech Recognition (ASR)ClusteringSpeaker Identificationspeech-recognition+1Probing Speaker-specific Features in Speaker Representations
This study explores speaker-specific features encoded in speaker embeddings and intermediate layers of speech self-supervised learning (SSL) models. By utilising a probing method, we analyse features such as pitch, tempo…
Self-Supervised LearningSpeaker VerificationSpeech Synthesistext-to-speech+2Unsupervised Learning of Disentangled Speech Content and Style Representation
We present an approach for unsupervised learning of speech representation disentangling contents and styles. Our model consists of: (1) a local encoder that captures per-frame information; (2) a global encoder that captu…
DecoderSpeaker Recognition