Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Speaker DiarizationSpeech RecognitionSpeech SeparationSimilar Papers 제목 키워드 기반
EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting. Our proposed framework integrates speaker diarization based on end-to-end neura…
Decoderspeaker-diarizationSpeaker DiarizationSpeech SeparationMulti-channel Conversational Speaker Separation via Neural Diarization
When dealing with overlapped speech, the performance of automatic speech recognition (ASR) systems substantially degrades as they are designed for single-talker speech. To enhance ASR performance in conversational or mee…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Speaker Separationspeech-recognition+1Spatial-Temporal Activity-Informed Diarization and Separation
A robust multichannel speaker diarization and separation system is proposed by exploiting the spatio-temporal activity of the speakers. The system is realized in a hybrid architecture that combines the array signal proce…
speaker-diarizationSpeaker DiarizationSpeaker SeparationOnline speaker diarization of meetings guided by speech separation
Overlapped speech is notoriously problematic for speaker diarization systems. Consequently, the use of speech separation has recently been proposed to improve their performance. Although promising, speech separation mode…
Action DetectionActivity Detectionspeaker-diarizationSpeaker Diarization+1Separation Guided Speaker Diarization in Realistic Mismatched Conditions
We propose a separation guided speaker diarization (SGSD) approach by fully utilizing a complementarity of speech separation and speaker clustering. Since the conventional clustering-based speaker diarization (CSD) appro…
Clusteringspeaker-diarizationSpeaker DiarizationSpeech Separation