PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning.
Code (1)
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speaker-diarizationSpeaker Diarizationspeech-recognitionSpeech RecognitionSpeech SeparationSimilar Papers 제목 키워드 기반
TalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024
This paper describes the submissions of team TalTech-IRIT-LIS to the DISPLACE 2024 challenge. Our team participated in the speaker diarization and language diarization tracks of the challenge. In the speaker diarization …
speaker-diarizationSpeaker DiarizationSpeech SeparationEEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers
In this paper, we present a novel framework that jointly performs three tasks: speaker diarization, speech separation, and speaker counting. Our proposed framework integrates speaker diarization based on end-to-end neura…
Decoderspeaker-diarizationSpeaker DiarizationSpeech SeparationOne model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition
This paper presents a novel framework for joint speaker diarization (SD) and automatic speech recognition (ASR), named SLIDAR (sliding-window diarization-augmented recognition). SLIDAR can process arbitrary length inputs…
AllAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Decoder+4Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained superv…
Decoderspeaker-diarizationSpeaker DiarizationUnifying Diarization, Separation, and ASR with Multi-Speaker Encoder
This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (AS…
Speaker DiarizationSpeech RecognitionSpeech Separation