Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture
We propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates the strengths of memory-aware multi-speaker embedding (MA-MSE) and sequence-to-sequence (Seq2Seq) architecture, leading to improvement in both efficiency and performance. Next, we further decrease the memory occupation of decoding by incorporating input features fusion and then employ a multi-head attention mechanism to capture features at different levels. NSD-MS2S achieved a macro diarization error rate (DER) of 15.9% on the CHiME-7 EVAL set, which signifies a relative improvement of 49% over the official baseline system, and is the key technique for us to achieve the best performance for the main track of CHiME-7 DASR Challenge. Additionally, we introduce a deep interactive module (DIM) in MA-MSE module to better retrieve a cleaner and more discriminative multi-speaker embedding, enabling the current model to outperform the system we used in the CHiME-7 DASR Challenge. Our code will be available at https://github.com/liyunlongaaa/NSD-MS2S.
Code (1)
Tasks
speaker-diarizationSpeaker DiarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Speaker Diarization with Mixture of Experts
In this paper, we propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates a memory-aware multi-speaker embedding mo…
Mixture-of-Expertsspeaker-diarizationSpeaker DiarizationSemi-supervised multi-channel speaker diarization with cross-channel attention
Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios. This paper proposes a semi-supervised speaker diarization system to utilize la…
speaker-diarizationSpeaker DiarizationTOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization
Recently, end-to-end neural diarization (EEND) is introduced and achieves promising results in speaker-overlapped scenarios. In EEND, speaker diarization is formulated as a multi-label prediction problem, where speaker a…
speaker-diarizationSpeaker DiarizationVocal Bursts Valence PredictionCommunity Detection Graph Convolutional Network for Overlap-Aware Speaker Diarization
The clustering algorithm plays a crucial role in speaker diarization systems. However, traditional clustering algorithms suffer from the complex distribution of speaker embeddings and lack of digging potential relationsh…
ClusteringCommunity DetectionGraph Generationspeaker-diarization+1Diarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models
Multi-speaker automatic speech recognition (MS-ASR) faces significant challenges in transcribing overlapped speech, a task critical for applications like meeting transcription and conversational analysis. While serialize…
Automatic Speech Recognitionspeaker-diarizationSpeaker Diarizationspeech-recognition+1